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THE MERIT REVIEW PROCESS: 
ENSURING LIMITED FEDERAL RESOURCES 
ARE INVESTED IN THE BEST SCIENCE 


TUESDAY, JULY 26, 2011 

House of Representatives, 
Subcommittee on Research and Science Education, 

Committee on Science, Space, and Technology, 

Washington, DC. 

The Subcommittee met, pursuant to call, at 10:01 a.m., in Room 
2318 of the Rayburn House Office Building, Hon. Mo Brooks 
[Chairman of the Subcommittee] presiding. 


( 1 ) 
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HEAHING CHARTER 

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 

SUBCOMMITTEE ON RESEARCH AND SCIENCE 
EDUCATION 

U.S. HOUSE OF REPRESENTATIVES 

The Merit Review Process: 

Ensuring Limited Federal Resources Are Invested 
in the Best Science 

TUESDAY, JULY 26, 2011 
io:oo A.M.-i2:oo p.m. 

2318 RAYBURN HOUSE OFFICE BUILDING 


Purpose 

On Tuesday, July 26, 2011, the Subcommittee on Research and Science Education 
will hold a hearing to examine the merit review grant award process and its effect 
on federally funded scientific research, in an effort to understand the strengths and 
potential weaknesses of the process. 

Witnesses 

• Dr. Cora Marrett, Deputy Director, National Science Foundation 

• Dr. Keith Yamamoto, Vice Chancellor for Research, University of California San 
Francisco 

• Dr. Nancy Jackson, President, American Chemical Society 

• Dr. Jorge Jose, Vice President for Research, Indiana University 

Overview 

• A number of federal agencies, from the Department of Energy to the National 
Institutes of Health and the National Science Foundation, use various types of peer 
or merit review to evaluate proposals and make recommendations to award federal 
funds. 

• The National Science Foundation (NSF) has three funding mechanisms: grants, 
cooperative agreements, and contracts. NSF makes merit-based grant awards to re- 
searchers, educators, and students. In Fiscal Year 2010 (FY 10), NSF received 
55,542 proposals and awarded 12,996 grants, a 23 percent funding rate. Fifty per- 
cent of its budget was devoted to new and continuing grants. ^ 

• Approximately 96 percent of NSF grant proposals are evaluated through an ex- 
ternal review process, commonly known as the NSF merit review process. The proc- 
ess utilizes subject matter experts to review proposals through the mail, in-person 
at a panel review, or through a combination of both (early-concept grants, rapid re- 
sponse grants, and small conferences and workshops are evaluated through an in- 
ternal merit review process). 

• The NSF merit review process evaluates proposals based on two criteria, intel- 
lectual merit and broader impacts. A National Science Board Task Force is cur- 
rently examining the two criteria and a report is forthcoming. Since 2007, NSF has 
also been promoting potentially transformative concepts through additional lan- 
guage added to the intellectual merit criteria. 

Background 

Federal funding is disbursed in a number of ways, including through contracts, 
cooperative agreements and grants. The process by which many federal agencies 
evaluate potential grant awards is often termed merit or peer review. This process 
can take several different forms or utilize different processes; however, in general. 


1 Report to the National Science Board on the National Science Foundation’s Merit Review 
Process Fiscal Year 2010, p. 7. 
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it requires that the grant proposals be reviewed and evaluated by subject matter 
experts not associated with the proposal. 

The National Science Foundation (NSF) uses grants for the majority of its funding 
disbursements. There are two basic grants. A standard award has a duration of 
typically one 09five years, but is fully funded in the first fiscal year. A continuing 
grant, also for a multi-year project, is provided in annual increments. The first year 
of funding for a continuing grant comes with a statement of intent to continue the 
funding with continuing grant increments (CGIs) through completion of the project, 
but the continuation is contingent on whether NSF deems satisfactory progress, 
availability of funds, and the receipt and approval of annual reports. 

Cooperative agreements are used when the project requires substantial agency in- 
volvement like research centers and multi-user facilities. 


Percentage of NSF Awards by Funding Mechanism 


CATEGORY 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

2010 

Standard Grants 

25% 

25% 

23% 

25% 

26% 

28% 

44% 

37% 

New Continuing 

16% 

14% 

14% 

13% 

14% 

13% 

8% 

13% 

CGIs and Supplements 

26% 

28% 

29% 

28% 

26% 

26% 

18% 

18% 

Cooperative Agreements 

25% 

24% 

24% 

23% 

22% 

23% 

21% 

23% 

Other* i 

9% 

9% 

10% 

11% 

11% 

11% 

9% 

9% 


*lncludes contracts, fellowships, interagency agreements, and IPA agreements 

Source: NSF Enterprise Information System 12/22/10. Percentages may not sum to 100 due to rounding. 


NSF utilizes an internal merit review process for a fraction of its grant awards, 
including the Early-concept Grants for Exploratory Research (EAGER) and Grants 
for Rapid Response Research (RAPID). ^ However, the bulk of NSF funded grants 
are evaluated through an external grant review process, known as the NSF merit 
review process. 

NSF Merit Review Process 

Grant proposals are required to be submitted electronically. NSF program officers 
ensure each proposal has been assigned to the correct office for review and deter- 
mine the appropriate level of review (internal or external). Proposals are returned 
without being reviewed if they do not comply with NSF regulations, including sepa- 
rately addressing the intellectual merit and broader impacts criteria. 

Program officers choose proposal reviewers and panel members from an NSF 
database of over 300,000 reviewers. They can also recruit reviewers based on lit- 
erature searches, professional activities, and other reviewer recommendations. In 
addition, they also screen all reviewers for potential conflicts of interest and provide 
guidance and instructions. 

Reviewers provide comments by mail or through the meeting of a panel session 
(these are most often in person but panelists may also meet virtually). Once review- 
ers return comments, or a panel convenes, program officers are responsible for syn- 
thesizing comments and recommending the award or decline of each proposal. Re- 
viewers provide narratives and categorical ratings which the program officer takes 
into account. An “Excellent” rating does not guarantee the award of funding. In FY 
2010, 3,743 proposals that received an average review of “Excellent” were funded 
and 1,312 were not, and 4,560 proposals that received an average review of “Very 
Good to Excellent” were funded while 6,318 were not. ® 


2 Early-concept Grants for Exploratory Research (EAGER) are two-year awards for up to 
$300,000. These awards support quick-response research, or research having a severe urgency 
with regard to availability of data, facilities, or equipment. 

3 Report to the National Science Board on the National Science Foundation’s Merit Review 
Process Fiscal Year 2010, p. 32. 
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Distribution of Average Reviewer Ratings for 
Awards and Declines, FY 2010 



■ Awards 

■ Declines 


No Score Poor to Fair Fair to Good Good to Very Good Excellent 
Very Good to Excellent 


Source: NSF Enterprise Information System 10/01/10 


The Division Director reviews the program officer’s recommendations and then 
passes them on to the Division of Grants and Awards, who vets the eligibility of 
the awardee, negotiates any necessary changes, and disburses the award. The Direc- 
tor’s Review Board reviews any award in excess of 2.5 percent of the awarding Divi- 
sion’s budget. The National Science Board (NSB) must approve any award in excess 
of $3 million dollars, or one percent or more of the awarding Directorate’s prior year 
current plan, whichever is greater. 

All those who submitted proposals receive notification as to whether or not an 
award will be made. Those to whom funding was declined receive copies of the re- 
views as well as information on the number of grants awarded and the number of 
proposals in each category. If a proposal is declined, the proposer may ask the pro- 
gram officer for further clarification. If he is still unsatisfied, he may make a recon- 
sideration request to the relevant Assistant Director and a second request to the 
Deputy Director. (See Appendix A for the NSF Flow Chart.) 

Types of Review 

Proposals submitted to the merit review system are reviewed in three ways. 
Through “mail-only” reviews, proposals are sent to reviewers who are asked to sub- 
mit written comments to NSF. Through “panel-only” reviews, reviewers serve on in- 
person (or virtual) panels to discuss reviews and provide advice to the program offi- 
cer. Additionally, some proposals receive a combination of mail and panel review, 
which can take place in a number of ways. 

There is value in each type of review. Mail review allows for better matching be- 
tween the expertise of reviewers and proposals. Panel review allows for interplay 
between reviewers in the evaluation of proposals and the integration of different 
perspectives in the review of proposals. According to NSF, “Using panels in the re- 
view process tends to reduce proposal processing time (time-to-decision), compared 
to mail-only reviews. For example, in FY 10, 78 percent of all proposals reviewed 
by panel-only were processed within six months, compared to 72 percent for mail 
plus panel and 55 percent for mail-only.” * While in-person panels are most com- 
mon, “virtual panels” are being convened more often. Virtual panels allow reviewers 
to participate from remote locations using interactive technology. The combination 
of mail and panel reviews “is used frequently because it combines the in-depth ex- 
pertise of mail review with the comparative analysis of panel review.” ® 


Report to the National Science Board on the Nationl Science Foundation’s Merit Review 
Process Fiscal Year 2010, page 29. 
s Ibid. 
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The Program Officer 

NSF program officers are made up of permanent (54 percent) and non-permanent 
(46 percent) employees; ® all are subject matter experts in the areas they manage 
with advanced degrees or credentials. “Some non-permanent program officers are ‘on 
loan’ as ‘Visiting Scientists, Engineers, and Educators’ (VSEEs) for up to three years 
from their host institutions. Others are supported through grants to the home insti- 
tutions under the terms of the Intergovernmental Personnel Act (IPA).” These “ro- 
tators” ensure that new and fresh scientific ideas and specialties come through the 
Foundation and help to prevent institutional or innovative stagnation. One draw- 
hack, however, is the loss of institutional knowledge when a rotator leaves and the 
challenge of frequently bringing new rotators up to speed on NSF policies and proc- 
esses. 

The Foundation expects program officers to administer balanced portfolios and re- 
quires them to utilize the advice and expertise of the proposal reviewers while as- 
sessing proposals in terms of each portfolio. In order to create a balanced portfolio, 
program officers are expected to additionally evaluate proposals for a number of cri- 
teria, including, but not limited to: geo^aphic distribution; novel approaches to sig- 
nificant research questions; capacity building in a new and promising research area; 
potential impact on the development of human resources and infrastructure; and 
NSF core strategies, including integration of research and education, broadening 
participation, and promoting partnerships. ® 

Merit Review Criteria 

Since initial approval in 1997, every NSF grant proposal has been reviewed based 
on two merit review criteria, intellectual merit and broader impacts. While addi- 
tional consideration may be given for a number of reasons including special require- 
ments of the program, intellectual merit and broader impacts remains the corner- 
stone of the NSF merit review process. 

• Intellectual Merit. What is the intellectual merit of the proposed activity? How 
important is the proposed activity to advancing knowledge and understanding with- 
in its own field or across different fields? How well qualified is the proposer (indi- 
vidual or team) to conduct the project? (If appropriate, the reviewer will comment 
on the quality of prior work.) To what extent does the proposed activity suggest and 
explore creative, original, or potentially transformative concepts? How well con- 
ceived and organized is the proposed activity? Is there sufficient access to resources? 

• Broader Impacts. What are the broader impacts of the proposed activity? How 
well does the activity advance discovery and understanding while promoting teach- 
ing, training, and learning? How well does the proposed activity broaden the partici- 
pation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, 
etc.)? To what extent will it enhance the infrastructure for research and education, 
such as facilities, instrumentation, networks and partnerships? Will the results be 
disseminated broadly to enhance scientific and technological understanding? What 
may be the benefits of the proposed activity to society? ® 

The America COMPETES Reauthorization Act of 2010 expands the broader im- 
pacts criteria to include activities to achieve the following goals: (1) increase the eco- 
nomic competitiveness of the United States; (2) development a globally competitive 
STEM workforce; (3) increase participation of women and underrepresented minori- 
ties in STEM; (4) increase partnerships between academia and industry; (5) improve 
pre-K 0912 STEM education and teacher development; (6) improve undergraduate 
STEM education; (7) increase public scientific literacy; and (8) increase national se- 
curity. 

In February 2010, the NSB reconstituted the Task Force on Merit Review. The 
Board charged the 'Task Force with “examining the two Merit Review Criteria and 
their effectiveness in achieving the goals for NSF support for science and engineer- 
ing research and education.” 


® Report to the National Science Board on the Nationl Science Foundation’s Merit Review 
Process Fiscal Year 2010, page 34. 

7 Ibid. 

® Report to the National Science Board on the Nationl Science Foundation’s Merit Review 
Process Fiscal Year 2010, p. 26. 

^ Report to the National Science Board on the Nationl Science Foundation’s Merit Review 
Process Fiscal Year 2010, pp. 21 0922. 

u’ NSB Task Force on Merit Review, http: ! I www.tisf.gov !nsh ! committees j 
tskforce -mr- charge .j sp . 



7 


In June 2011, after a year of review, NSB and NSF put out a call for public com- 
ment (closed July 14, 2011) on proposed revisions to the merit review criteria. The 
proposed changes maintain the themes of intellectual merit and broader impacts 
while establishing key principles of the merit review criteria. The proposed changes 
include the identification of national goals which every NSF project should seek to 
advance, including but not limited to: the increased economic competitiveness of the 
United States; the increased participation of women, persons with disabilities, and 
underrepresented minorities in STEM; the increased public scientific literacy and 
public engagement with science and technology; and increased national security. 
(See Appendix B for the complete proposal.) 

Potential Challenges 

While the NSF merit review process is widely considered the most effective of its 
type for the awarding of federal funding, there are existing challenges to be consid- 
ered in an effort to strengthen the process. Questions remain about the way in 
which scientific priorities are established and whether the process is truly sup- 
porting innovative research and researchers. Below are some additional challenges: 

• Transformative Research Research that questions existing science often faces 
additional hurdles when facing review by scientific researchers in that field, espe- 
cially during lean economic times as experts favor more conservative funding ap- 
proaches. Since 2007, NSF has been working to ensure transformative research is 
considered appropriately and such proposals are provided an opportunity to compete 
through the merit review process, including adding explicit language in the intellec- 
tual merit criteria for transformative concepts. EAGER grants are intended to be 
used, in part, to fund potentially transformative ideas for which there is little to no 
preliminary data and, as such, would fare poorly in the standard merit review proc- 
ess. NSE has also incorporated efforts to encourage transformative research in its 
training of program officers and reviewers. It is also experimenting with modifica- 
tions in the review process to help identify transformative proposals. Are these ef- 
forts working? Is there more to be done, within the process, to encourage trans- 
formative science? 

• Ensuring a Pipeline for U.S. Students by Encouraging New Principal Investiga- 
tors New Principal Investigators (Pis) often do not have the same level of experience 
or access to resources that established Pis have, both considerations included as 
part of the intellectual merit criteria. In FY 10, new Pis submitted 21,545 proposals 
and received 3,620 awards, a funding rate of 17 percent; prior Pis were funded at 
a rate of 28 percent, A strong system properly encourages new investigators to 
participate in the scientific arena in order to ensure a pipeline for U.S. student par- 
ticipation in scientific endeavors. The Faculty Early Career Development (CAREER) 
Program offers specific funding opportunities for new Pis to help in this endeavor, 
but is this enough? Does the merit review process encourage the participation of 
new Pis? 

• Parity for Institutions Institutions that are not regular grant recipients do not 
always have the same resources or proficiencies as those institutions that consist- 
ently receive federal funding. “For FY 10, the average funding rate was 26 percent 
for the top 100 Ph.D. -granting institutions (classified according to the amount of FY 
10 funding received). In comparison, the rate was 17 percent for Ph.D. -granting in- 
stitutions that are not in the top 100 NSF-funded category.” Are those institu- 
tions, not regularly in receipt of federal funding, encouraged to submit grant pro- 
posals and participate in the merit review process? 

• Multidisciplinary Review As NSF seeks to grow its multidisciplinary projects, 
the merit review process must consider the management of reviews that incorporate 
a combination of scientific disciplines in order to fund the strongest multidisci- 
plinary proposals. Is the current process able to effectively encourage and evaluate 
multidisciplinary projects? 

In all, the merit review process must continue to balance these challenges with 
the inherent need to fund the strongest science. 


Report to the National Science Board on the National Science Foundations’s Merit Review 
Process Fiscal Year 2010, p. 8. 

Report to the National Science Board on the National Science Foundations’s Merit Review 
Process Fiscal Year 2010, pp. 10 0911. 
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APPENDIX A: NSF Merit Review Process Flow Chart 
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Source; National Science Foundation 
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APPENDIX B: Merit Review Principles and Criteria* 

The identification and description of the merit review criteria are firmly grounded in the following principles: 

1 . All NSF projects should be of the highest intellectual merit with the potential to advance the frontiers of 
knowledge. 

2. Collectively, NSF projects should help to advance a broad set of important national goals, including: 

o Increased economic competitiveness of the United States, 
o Development of a globally competitive STEM workforce. 

o Increased participation of women, persons with disabilities, and underrepresented minorities in 
STEM. 

o Increased partnerships between academia and industry, 
o Improved pre-K-12 STEM education and teacher development, 
o Improved undergraduate STEM education. 

o Increased public scientific literacy and public engagement with science and technology, 
o Increased national security. 

o Enhanced infrastructure for research and education, including facilities, instrumentation, networks 
and partnerships. 

3. Broader impacts may be achieved through the research itself, through activities that are directly related to 
specific research projects, or through activities that are supported by the project but ancillary to the 
research. All are valuable approaches for advancing important national goals. 

4. Ongoing application of these criteria should be subject to appropriate assessment developed using 
reasonable metrics over a period of time. 

Intellectual merit of the proposed activity 

The goal of this review criterion is to assess the degree to which the proposed activities will advance the frontiers of 
knowledge. Elements to consider in the review are: 

1 . What role does the proposed activity play in advancing knowledge and understanding within its own field 
or across different fields? 

2. To what extent does the proposed activity suggest and explore creative, original, or potentially 
transformative concepts? 

3. How well conceived and organized is the proposed activity? 

4. How well qualified is the individual or team to conduct the proposed research? 

5. Is there sufficient access to resources? 

Broader impacts of the proposed activity 

The purpose of this review criterion is to ensure the consideration of how the proposed project advances a national 
goal(s). Elements to consider in the review are: 

1. Which national goal (or goals) is (or are) addressed in this proposal? Has the PI presented a compelling 
description of how the project or the PI will advance that goal(s)? 

2. Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or 
institutional engagement? 

3. Is the rationale for choosing the approach well-justified? Have any innovations been incorporated? 

4. How well qualified is the individual, team, or institution to carry out the proposed broader impacts 
activities? 

5. Are there adequate resources available to the PI or institution to carry out the proposed activities? 


*Source; National Science Board, http://wv.\v.nsf,gov/nsb/publications/201 1/06 mitfisp 
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Chairman Brooks. The Subcommittee on Research and Science 
Education will come to order. 

Good morning. Welcome to today’s hearing entitled “The Merit 
Review Process: Ensuring Limited Federal Resources Are Invested 
in the Best Science.” I am going to give an opening statement. 

Our hearing today presents us with an opportunity to examine 
the merit review process for awarding federal grant funds. It is our 
goal to highlight the benefits of the process, while acknowledging 
that no process involving human decision making is flawless. The 
focus of today’s hearing will primarily be on the merit review proc- 
ess at the National Science Foundation. 

The National Science Foundation Act of 1950 directs NSF to ini- 
tiate and support basic scientific research and programs to 
strengthen scientific research potential and science education pro- 
grams at all levels. NSF works to accomplish this Congressional di- 
rective through the issuance of merit-based awards to researchers, 
educators and students at approximately 1,900 U.S. colleges, uni- 
versities and other institutions. In 1994, the National Academies 
touted it as among the best procedures known for insuring the 
technical excellence of research projects that receive public support, 
but the process has changed since then, and we need to make sure 
that is still the case. 

As we know, a large number of potentially fundable proposals 
are declined each year. The Foundation received over 55,000 pro- 
posals for funding in fiscal year 2010 and funded nearly 13,000, or 
23 percent, of them. Many of the proposals received were not wor- 
thy of federal funding, but it is also true that many were not fund- 
ed because federal funds are limited. So given that those limited 
dollars should go to the very best scientific research, NSF must 
maintain a robust and transparent merit review process. 

Today, our witnesses will share their thoughts on how the proc- 
ess works and its strengths and weaknesses. We want to know if 
the current process spurs or stifles innovation, how award decisions 
are actually made after receiving peer review, and if there are 
flaws in the system that may be providing precious federal funds 
to lower rated proposals over more highly rated proposals. 

In exercising its oversight role, this Subcommittee must ensure 
that federal dollars are being spent on the best science. This exam- 
ination of the merit review process will help us to understand how 
programmatic funding decisions are made and how, in turn, those 
decisions interact with Foundation-wide priorities. 

I look forward to hearing from each of our witnesses on this im- 
portant topic. I thank you for joining us. 

[The prepared statement of Mr. Brooks follows:] 

Prepared Statement of Chairman Mo Brooks 

Good morning, and welcome to each of our witnesses. Our hearing today presents 
us with an opportunity to examine the merit review process for awarding federal 
grant funds. It is our goal to highlight the benefits of the process, while acknowl- 
edging that no process involving human decision making is flawless. The focus of 
today’s hearing will primarily be on the merit review process at the National 
Science Foundation (NSF). 

The National Science Foundation Act of 1950 directs NSF to “to initiate and sup- 
port basic scientific research and programs to strengthen scientific research poten- 
tial and science education programs at all levels.” NSF works to accomplish this 
Congressional directive through the issuance of merit-based awards to researchers, 
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educators and students at approximately 1,900 U.S. colleges, universities, and other 
institutions. In 1994, the National Academies touted it as being among “the best 
procedures known for insuring the technical excellence of research projects that re- 
ceive public support,” but the process has changed since then, and we need to make 
sure that is still the case. 

As we know, a large number of potentially fundable proposals are declined each 
year. The Foundation received over 55,000 proposals for funding in Fiscal Year 2010 
and funded nearly 13,000, or 23 percent, of them. Many of the proposals received 
were not worthy of federal funding, but it is also true that many were not funded 
because federal funds are limited. So, given that those limited dollars should go to 
the very best scientific research, NSF must maintain a robust and transparent 
merit review process. 

Today, our witnesses will share their thoughts on how the process works and its 
strengths and weaknesses. We want to know if the current process spurs or stifles 
innovation, how award decisions are actually made after receiving peer review, and 
if there are flaws in the system that may be providing precious federal funds to 
lower rated proposals over more highly rated proposals. 

In exercising its oversight role, this Subcommittee must ensure that federal dol- 
lars are being spent on the best science. This examination of the merit review proc- 
ess will help us to understand how programmatic funding decisions are made and 
how, in turn, those decisions interact with Foundation-wide priorities. 

I look forward to hearing from each of our witnesses on this important topic; 
thank you for joining us. 

Chairman Brooks. At this point, I recognize Mr. Lipinski, who 
is the Ranking Member of this Subcommittee. 

Mr. Lipinski. Thank you, Chairman Brooks, and thank you for 
holding the hearing this morning. 

It will not surprise you when I say that I have had a great inter- 
est in this topic since I first submitted a grant proposal to the Na- 
tional Science Foundation when I was a graduate student. That in- 
terest continued throughout my academic career, and when I was 
elected to Congress I joined what was then the Science and Tech- 
nology Committee to a large extent — actually it was the Science 
Committee at the time when I joined — to a large extent because of 
my strong belief in the NSF and its mission. 

I agree with the statements of all of the witnesses here today 
that NSF’s merit review system remains the gold standard for the 
world. And I don’t say that just because that NSF grant application 
I submitted when I was in grad school was successful. 

At the same time, I recognize that there are challenges in any 
system for allocating limited research dollars. I agree with Chair- 
man Brooks that it is our job on this Subcommittee to hold hear- 
ings such as this one to discuss these challenges and collectively 
imagine how we might continue to make NSF, and the merit-re- 
view system that it manages, even stronger. Particularly in this 
tight budget environment, it is incumbent upon us all to make sure 
that the system for funding excellent science is as efficient and ef- 
fective as possible. 

In 2009, when I was Chair of this Subcommittee, we looked at 
a few slices of the broader topic being addressed today when we 
held a hearing on high-risk, high-reward, or what we also call 
transformative research. Dr. Yamamoto was not on the panel for 
that hearing, but a report that he helped author, the ARISE report, 
played a central role and remains relevant today. I look forward to 
learning from his deep expertise on this topic. 

But there are many issues that we have not examined in detail, 
including the extent to which faculty from lesser-resourced institu- 
tions face an uneven playing field. I am also interested in the ex- 
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tent to which the institutional structures underlying NSF’s peer-re- 
view system influence decisions and the benefits and drawbacks of 
different approaches to peer review. 

For example, I am intrigued by proposals to conduct committee 
review in virtual environments such as Second Life. While I am 
perhaps a little skeptical, I recognize that virtual review has the 
potential to save taxpayers a lot of money in travel expenses, as 
well as broaden the pool of reviewers. So I am glad that the NSF 
is looking into innovative approaches. But we need to be confident 
that the group dynamics in a virtual environment, while certain to 
be different, do not in any way undermine the quality of NSF merit 
review. 

I am particularly interested in hearing the panel’s recommenda- 
tions about some of the alternatives to standard merit review, not 
as a replacement of, but rather as a complementary approach. For 
example, in last year’s COMPETES Act, I authored a provision 
that authorizes prize programs at all of the science agencies. While 
NSF, as a basic research agency, would need to design and imple- 
ment a prize program that looks very different from those run by 
DARPA or NASA, I continue to believe that the NSF should experi- 
ment with some pilot projects to award research prizes. I look for- 
ward to hearing an update from Dr. Marrett on the Foundation’s 
thoughts on this subject. 

So I see this hearing as an opportunity to examine the real chal- 
lenges that do exist in a very strong review system and to discuss 
current and novel approaches to overcoming those challenges. 

Finally, Mr. Chairman, I think it is important to point out that 
the National Science Board is in the middle of a process to review 
and revise the existing merit review criteria. While I believe this 
hearing is important, it is critical that in the fall when the Board 
has finished its work and produced a new set of review criteria for 
the scientific community, this Committee should examine what the 
Board has come out with. Especially in light of the provision in last 
year’s COMPETES bill directing the agency to clarify the purpose 
and implementation of the broader impacts review criterion, it will 
be helpful at some point later this year or early next year to revisit 
this topic, and I certainly hope that we can do that. 

With that, Mr. Chairman, I thank all of the witnesses for being 
here today and I yield back. 

[The prepared statement of Mr. Lipinski follows:] 

Prepared Statement of Ranking Member Daniel Lipinski 

Thank you, Chairman Brooks, and thank you for holding this hearing this morn- 
ing. It will not surprise anyone when I say that I have had a great interest in this 
topic since I first submitted a grant proposal to the National Science Foundation 
as a graduate student. That interest continued throughout my academic career and 
when I was elected to Congress I joined what was then the Science and Technology 
committee to a large extent because of my strong belief in the NSF and its mission. 
I agree with the statements of all of the witnesses here today that NSF’s merit-re- 
view system remains the gold standard for the world. And I don’t say that just be- 
cause that NSF grant application I submitted when I was in grad school was suc- 
cessful. 

At the same time, I recognize that there are challenges in any system for allo- 
cating limited research dollars. I agree with Chairman Brooks that it is our job, on 
this Subcommittee, to hold hearings such as this one to discuss these challenges and 
collectively imagine how we might continue to make NSF, and the merit-review sys- 
tem that it manages, even stronger. Particularly in this tight budget environment 
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it is incumbent upon us all to make sure that the system for funding excellent 
science is as efficient and effective as possible. 

In 2009, when I was chair of this Subcommittee, we looked at a few slices ofthe 
broader topic being addressed today when we held a hearing on high-risk, high-re- 
ward, or what we also call transformative research. While Dr. Yamamoto was not 
on the panel for that hearing, a report that he helped author, the ARISE report, 
played a central role and remains relevant today. I look forward to learning from 
his deep expertise on this topic. 

But there are many issues that we have not examined in detail, including the ex- 
tent to which faculty from lesser resourced institutions face an uneven playing field. 
I am also interested in the extent to which the institutional structures underlying 
NSF’s peer-review system influence decisions and the benefits and drawbacks of dif- 
ferent approaches to peer review. 

For example, I am intrigued by proposals to conduct committee review in virtual 
environments such as Second Life. While I am perhaps a little skeptical, I recognize 
that virtual review has the potential to save taxpayers a lot of money in travel ex- 
penses, as well as to broaden the pool of reviewers. So I’m glad that the NSF is 
looking into innovative approaches. But we need to be confident that the group dy- 
namics in a virtual enviromnent, while certain to be different, do not in any way 
undermine the quality of NSF merit review. 

I’m particularly interested in hearing the panel’s recommendations about some of 
the alternatives to standard merit review, not as a replacement of, but rather as 
a complementary approach. For example, in last year’s COMPETES Act, I authored 
a provision that authorizes prize programs at all ofthe science agencies. While NSF, 
as a basic research agency, would need to design and implement a prize program 
that looks very different from those run by DARPA or NASA, I continue to believe 
that the NSF should experiment with some pilot projects to award research prizes. 
I look forward to hearing an update from Dr. Marrett on the Foundation’s thoughts 
on this subject. 

So I see this hearing as an opportunity to examine the real challenges that do 
exist in a very strong review system and to discuss current and novel approaches 
to overcoming those challenges. 

Finally, Mr. Chairman (as you already mentioned), I think it’s important to point 
out that the National Science Board is in the middle of a process to review and re- 
vise the existing merit review criteria. While I believe this hearing is critically im- 
portant, it is unfortunate that we couldn’t wait until this fall when the Board has 
finished its work and produced a new set of review criteria for the scientific commu- 
nity and this Committee to examine. Especially in light of the provision in last 
year’s COMPETES bill directing the agency to clarify the purpose and implementa- 
tion of the Broader Impacts Review Criterion, it will he helpful at some point later 
this year or next year to revisit this topic. 

With that, Mr .Chairman, I thank all of the witnesses for being here today and 
I 3deld back. 

Chairman Brooks. Thank you, Mr. Lipinski. 

At this time I would like to introduce our witnesses for today’s 
hearing. First, we have Dr. Cora B. Marrett, the Deputy Director 
of the National Science Foundation. Since January 2009, she has 
served as NSF’s Acting Deputy Director and Senior Advisor until 
her confirmation as Deputy Director in May 2011. Dr. Marrett 
holds a bachelor of arts degree from Virginia Union University, a 
master of arts and a doctorate from the University of Wisconsin- 
Madison, all in sociology. 

Dr. Keith Yamamoto is Vice Chancellor for Research at the Uni- 
versity of California at San Francisco. Dr. Yamamoto served on 
grant review panels for the NSF Biology Directorate, and as an ad 
hoc member of the National Science Board Taskforce on Trans- 
formative Research. 

Dr. Nancy Jackson serves as President of the American Chemical 
Society. Dr. Jackson is employed at Sandia National Laboratories 
and earned her Ph.D. in chemical engineering from the University 
of Texas at Austin. 
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And Dr. Jorge Jose is the Vice President for Research at Indiana 
University. A theoretical physicist and neurohiologist, Dr. Jose re- 
ceived his undergraduate, master’s and Ph.D. in the Department of 
Physics at the National Autonomous University of Mexico. 

As our witnesses should know, spoken testimony is limited to 
five minutes each, after which the members of the committee will 
have five minutes each to ask questions. 

I now recognize our first witness. Dr. Cora Marrett, for five min- 
utes. 


STATEMENT OF DR. CORA MARRETT, 

DEPUTY DIRECTOR, NATIONAL SCIENCE FOUNDATION 

Dr. Marrett. Thank you. Chairman Brooks, Ranking Member, 
Lipinski, and the distinguished Members of the Subcommittee 
staff. I am pleased to have this opportunity to discuss the National 
Science Foundation’s merit review process. 

For over 60 years, NSF has been a steward of the Nation’s 
science and engineering enterprise with a proven track record for 
producing results. Despite its relatively small size, NSF has an im- 
portant impact on scientific and engineering knowledge and aca- 
demic capacity. NSF’s investments in discovery, learning and inno- 
vation have been important to increasing America’s competitive- 
ness, our economic strength, national security and overall quality 
of life. 

NSF relies on a merit-based competitive process that is critical 
to fostering the highest standards of excellence and accountability. 
The process lies at the heart of the agency’s strategy for accom- 
plishing its overall mission. 

Now, of the 256 American Nobel Prize recipients in science since 
NSF first began to award research grants, of those 256, approxi- 
mately 75 percent have received NSF funding at some point in 
their careers. That includes 19 individuals in the last five years. 

The agency has a strong record of funding the most insightful 
ideas and visionary investigators. Our model of high-quality merit 
review has been emulated and replicated by other nations. Our 
merit review process helps assure that awards made by NSF are 
of the highest quality, are relevant to our goals and objectives and 
have an appropriate balance for the resulting portfolio. This assur- 
ance is critical as nearly 90 percent of NSF’s funding is allocated 
through merit review, allocated either as grants or cooperative 
agreements. 

As you have already noted, in the fiscal year 2010, NSF evalu- 
ated over 55,000 proposals through this process and made approxi- 
mately 13,000 new awards. This entailed conducting about 287,000 
proposal reviews and engaging nearly 46,000 members of the 
science and engineering community as reviewers. Underlying these 
statistics is a strategy that helps ensure that each proposal sub- 
mitted to NSF is reviewed in a fair, competitive, transparent, and 
in-depth manner. 

First, our program officers are experts in the scientific and engi- 
neering programs they manage. They hold strong credentials, usu- 
ally a Ph.D. or equivalent in a STEM field, and their funding rec- 
ommendations are subject to several additional layers of review. 
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Each proposal submitted to NSF is reviewed by these in-house ex- 
perts, and they in turn rely extensively on external experts to keep 
NSF-funded research at the frontier. As we emphasize, this is a 
unique aspect of NSF, the knowledge of our program officers, who 
are then empowered to make recommendations on funding. 

Each proposal must meet the highest standards in terms of two 
merit review criteria: intellectual merit and broader impacts. Intel- 
lectual merit encompasses the potential of the research to advance 
knowledge, the originality and creativity of the proposed activity 
and the qualifications of the researchers. Broader impacts include 
technological innovation, societal benefits and opportunities to in- 
clude a diversity of participants. 

Finally, the merit review process itself is constantly assessed 
through evaluations by committees of visitors, advisory committees 
and other stakeholders. NSF continuously strives to maintain and 
improve the quality and transparency of the process. 

So in summary, the National Science Foundation is dedicated to 
ensuring that the merit review process remains robust, rigorous, 
and beyond reproach. This process enables us to carry out our mis- 
sion. I appreciate, then, this opportunity to appear before the Sub- 
committee on this important topic and would be pleased to answer 
any questions you might have. 

[The prepared statement of Dr. Marrett follows:] 

Prepared Statement of Dr. Cora Marrett, 

Deputy Director, National Science Foundation 

Chairman Brooks, Ranking Member Lipinski, and distinguished Members of the 
Subcommittee, thank you for inviting me to participate in this hearing on “The 
Merit Review Process.” 

I am delighted to discuss the National Science Foundation’s (NSF) Merit Review 
Process with you. As you well know, NSF is the primary Federal agency supporting 
research at the frontiers of knowledge, across all fields of science and engineering 
(S&E) and all levels of S&E education. Its mission, vision and goals are designed 
to maintain and strengthen the vitality of the U.S. science and engineering enter- 
prise. As part of the overall national R&D enterprise, the basic research and edu- 
cation activities supported by NSF are vital to the economic advancement of the 
U.S. and provide the know-how that allows the U.S. to respond rapidly and effec- 
tively to a range of unexpected challenges. The NSF merit review process lies at the 
heart of the agency’s strategy for accomplishing its overall mission. As such, NSF 
is continuously striving to maintain and improve the quality and transparency of 
the process. 

Before I begin my discussion of the unique elements of the NSF merit review sys- 
tem, let me first describe the essential features of merit review writ large. In gen- 
eral, merit review refers to an independent assessment of a plan’s worthiness. The 
Code of Federal Regulations (Section 600.13 of title 10) defines Merit Review as a 
“thorough, consistent and objective examination of applications based on pre-estab- 
lished criteria by persons who are independent of those individuals submitting the 
applications and who are knowledgeable in the field of endeavor for which support 
is requested.” 

I would also like to note here that although the terms “merit review” and “peer 
review” are often used interchangeably, they are not equivalent terms. NSF made 
this distinction clear back in 1986, based on a report from an external Advisory 
Committee on Merit Review, established by then-director Erich Bloch at the request 
of the National Science Board. As is described by Marc Rothenberg, the NSF histo- 
rian, in his 2010 article “Making Judgments about Grant Proposals: A Brief History 
of the Merit Review Criteria at the National Science Foundation:” 

• According to the committee, the term “peer review” was properly a restrictive 
term referring to the evaluation of the technical aspect of the proposal. However, 
for more and more federally funded research, “technical excellence” was, in the 
words of the committee, “a necessary but not fully sufficient criterion for research 
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funding.” Acknowledging that the NSF (as well as other federal agencies) was using 
a wide range of nontechnical criteria as part of the decision-making process, the 
committee suggested that the term “merit review” more accurately described the 
NSF selection process. 

• The committee’s recommendation was accepted by Director Bloch, and since 
then NSF has used the term “merit review” to describe our process. 

Since its founding, NSF has relied on the merit review process to allocate the vast 
majority of its funding. As in other agencies, this has involved the use of proposals 
from prospective researchers that are judged on their merits by knowledgeable per- 
sons. But there are several elements that give merit review at the NSF its distinct 
features. For one, right from the beginning, NSF utilized the project grant mecha- 
nism (as opposed to a contract mechanism) for providing funds. This was a rather 
radical concept back in 1951, when most government operations used contracts. 
Since that time, the use of the grant mechanism has been adopted by many federal 
extramural research funding organizations. 

NSF’s process for deciding which proposals to fund differs from the approach of 
a number of other funding agencies and organizations (such as philanthropic foun- 
dations) nationally and internationally. Perhaps the most distinctive differences are 
our reliance on expertise from both outside and within the Foundation, and the dis- 
cretionary authority vested in the NSF program officer to make funding rec- 
ommendations. Unlike many philanthropic foundations (and even some federal re- 
search funding programs), NSF policy requires that the program officers seek exter- 
nal expert advice before making most of their funding recommendations. However, 
in contrast to a number of other funding bodies, the external reviewers do not make 
binding recommendations that the program officer is obliged to follow, although pro- 
gram officers always pay close attention to all external reviews. Because of the re- 
sponsibility we give our program officers, NSF sets a high standard for excellence 
in that position. Our program officers are subject matter experts in the scientific 
areas that they manage, and bring strong credentials with them, including advanced 
educational training (e.g., a Ph.D. or equivalent credentials) in science or engineer- 
ing, and deep experience in research, education, and/or administration. 

NSF has chosen to give the program officer the responsibility for making funding 
recommendations to enable a more strategic and long-term approach for building 
the award portfolio. As important as the input of the external scientific experts is, 
they have only a sn^shot view of the current set of proposals they are evaluating. 
The NSF program officer is responsible for putting that snapshot view into the larg- 
er context of the entire award portfolio they are managing, which can lead to a more 
diverse and robust portfolio overall. Together with the division directors, who have 
the authority to review and act on the program officers’ recommendations, program 
officer teams are poised to identify promising research that responds to national pri- 
orities identified by Congress and the Administration. In addition, program officers 
can incorporate agency or programmatic priorities, which are articulated in the an- 
nual agency budget, special solicitations, and standing program descriptions, all of 
which are available to the community via the NSF Web site. 

The NSF merit review process is described in full detail on the NSF Web site 
(http:! I www.nsf.gov I bfa I dias I policy I meritreview !). There is also a summary of the 
major steps in the merit review process in the annual Report to the National 
Science Board on the Merit Review Process (the most recent report covering activi- 
ties in FY 2010 can be found at http: II www.nsf.gov Insbl publications ! 2011 ! 
nsbll41.pdf). It is worth noting here that the key features of the NSF process have 
remained remarkably stable over time. Any changes that have been incorporated 
have sought primarily to clarify the process and make it more transparent. For ex- 
ample, initially only excerpts of the external reviews were shared with the proposal 
authors. Over time, NSF provided the verbatim reviews (but not the identities of 
the reviewers) to the applicant. Similarly, over time there have been modifications 
to the number and clarity of the review criteria. In the America COMPETES Reau- 
thorization Act, the broader impacts criterion is specifically mentioned, and the Na- 
tional Science Board is in the process of analyzing the many comments received on 
this topic. 

A flowchart that graphically depicts the major steps in the merit review process 
and a timeline is attached to this testimony as Appendix I. These steps include: 

• Assignment to the appropriate program for review. Principal investigators ini- 
tiate this process by selecting the program or programs to which they wish to sub- 
mit their proposal. Once submitted, the cognizant program officers for those pro- 
grams confirm that the assignment is appropriate. On occasion, a proposal may be 
reassigned to another program where there is a better fit. During this initial assign- 
ment process, it is not uncommon for proposals to be assigned to multiple programs 
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for review, if the subject is interdisciplinary in nature, or if the question is of inter- 
est and relevance to more than one program. 

• Administrative review of all proposals for compliance with NSF regulations. 
These regulations, which are intended to ensure fairness in the review process, are 
described in the Grant Proposal Guide, which is widely available to the NSF com- 
munity on the NSF Web site (http: ! Iwww.nsf gov ! pubs ! policydocs ! pappguide ! 
nsfllOOl Insfll -l.pdf). Proposals that do not comply with these regulations may be 
returned without review. 

• Merit review of all proposals that pass the administrative review. As noted 
above, a critical feature of NSF’s process is the use of both external review by ex- 
perts in the field and internal review by NSF’s corps of program officers. The pro- 
gram officers are responsible for administering the merit review process from begin- 
ning to end, starting with identifying and recruiting appropriate peer reviewers 
from the external community to serve either as individual reviewers for a particular 
proposal (referred to as “ad hoc” reviewers) or as members of a panel of reviewers 
who evaluate a larger set of proposals. To ensure that they receive substantive re- 
views from a variety of perspectives, the program officers reach out to a broad range 
of experts for input — in fiscal year 2010, over 46,000 external peer reviewers from 
academia, government, and occasionally industry provided authoritative advice to 
the Foundation. Selection of expert peer reviewers may be based on the program 
officer’s knowledge, references listed in the proposal, individuals cited in recent pub- 
lications or relevant journals, presentations at professional meetings, reviewer rec- 
ommendations, bibliographic and citation databases, or suggestions from the pro- 
posal author (subject to the program officer’s discretion). In making these selections, 
program officers pay very careful attention to avoiding conflicts of interest, both real 
and perceived. 

NSF takes seriously its responsibility to ensure that the merit review process is 
fair and equitable. One of the ways in which we address this responsibility is 
through the briefings that are given to each review panel before it begins its work. 
In these briefings, panelists are instructed on NSF’s review criteria (Intellectual 
Merit and Broader Impacts), and on maintaining confidentiality and avoiding con- 
flicts of interest. In addition, review panel briefings typically include alerting the 
reviewers to the phenomenon of implicit bias, which may adversely impact new in- 
vestigators, smaller institutions, and underrepresented groups. By guarding against 
the effects of implicit bias in the review process, NSF is working to ensure that 
there are equitable opportunities for all investigators. 

I should note here that while the vast majority of the proposals received at NSF 
(96%) are subject to both external and internal merit review, for some proposals the 
external review requirement is waived. This waiver provides necessary flexibility for 
handling proposals for which most of the external community would be conflicted 
(such as proposals for small conferences, workshops, or symposia), those for which 
there is a severe urgency (submitted through the Grants for Rapid Response Re- 
search, or RAPID, mechanism used, for example, on rapid-response research to the 
Deepwater Horizon oil spill), and those that request support for high-risk, poten- 
tially transformative exploratory work (submitted through the Early Grants for Ex- 
ploratory Research, or EAGER, mechanism). These proposals are usually only re- 
viewed internally by program officers with appropriate expertise. 

• Development of funding recommendations. A central tenet of the NSF merit re- 
view process is that the reviewer input is advisory in nature. Funding recommenda- 
tions are developed by the program officer, who is responsible for synthesizing the 
advice of the reviewers along with several other factors, with the goal of allocating 
funding to a diverse portfolio of projects that addresses a variety of considerations 
and objectives. In addition to their scientific expertise noted above, NSF program 
officers bring their own unique perspective born from their experience of working 
with hundreds, thousands, or — in some cases — tens of thousands of proposals. In de- 
veloping recommendations within the larger context of their overall portfolio, pro- 
gram officers consider carefully the individual merits of each proposal with respect 
to both its intellectual merit and the potential broader impacts of the project, and 
how each proposal might help advance a variety of portfolio goals such as: 

• Achieving special program objectives and initiatives; 

• Fostering novel approaches to significant research and education questions; 

• Building capacity in a new and promising research area; 

• Supporting high-risk proposals with potential for transformative advances; 

• Supporting NSF’s core strategies of integration of research and education 
and integrating diversity into NSF’s programs; 
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• Potential impact on human resources and infrastructure; 

• Other available funding sources; and 

• Geographic distribution. 

NSF has set a goal for completing this process within six months, from the time 
the proposal is submitted to the point at which the proposal is either declined or 
recommended for funding and forwarded to the Division of Grants and Agreements 
for the final stages of review and processing. The proposal assignment and adminis- 
trative review stage is typically complete within a few weeks. The bulk of the time 
is spent in the merit review stage, which can take three to four months to complete. 
Despite the volume of proposals that NSF receives annually (in FY 2010, over 
55,000 proposals were submitted, an increase of 23% over the previous year), NSF 
routinely processes the majority of these proposals (°75%) in fewer than six months. 

To ensure the integrity of the process, all program officer recommendations are 
reviewed by the division director (or other appropriate NSF official), who examines 
whether the process used to arrive at the decision has been executed in accordance 
with NSF’s policies and that the decision has been based on a thorough analysis 
of the merits of the proposal. Large awards may receive additional review, either 
by the Director’s Review Board (DRB) or additionally by the National Science Board 
(NSB). The DRB examines award recommendations with an average annual award 
amount of 2.5 percent or more of the awarding division’s prior year current plan. 
The NSB reviews recommended awards with an annual award amount of one per- 
cent or more of the awarding Directorate’s or Office’s prior year current plan, or less 
than one percent or more of the prior year total NSF budget at the enacted level. 
Once the funding recommendation is approved (at whatever level is appropriate), 
the Division of Grants and Agreements ensures that the award recommendation 
meets all of NSF’s requirements before officially issuing the award. 

In addition to having multiple layers of review of individual award recommenda- 
tions, NSF requires that all programs undergo an external review by Committees 
of Visitors (COVs) every three years. COV reviews provide NSF with external expert 
assessments of the quality and integrity of program operations and program-level 
technical and managerial matters pertaining to the merit review and final proposal 
decisions. Finally, retrospective analysis of the process is periodically performed on 
a Foundation-wide basis, including the statistical reports submitted to the NSB 
every year and the Impact of Proposal and Award Management Mechanisms 
(IPAMM) report of 2007 {http:! I www.nsf.gov I pubs 1 2007 / nsf0745 1 nsf0745.pdf). 

At the request of Congress, in 2005 the NSB undertook an examination of NSF’s 
Merit Review Process (http:llwww.nsf.govlnsbldocumentsl2005lnsb05119.pdf). 
The report concludes that: 

“The Board fully supports the current NSF system of merit review, which utilizes 
the peer review process as the principal driver in funding decisions. The Board also 
strongly endorses the role of NSF program officers’ discretionary authority, in con- 
currence with division directors, for ensuring the implementation and goals of both 
Merit Review Criteria, along with achieving a balanced portfolio of research and 
education awards, both within directorates and across the suite of NSF programs. 
Unlike a system based solely on peer reviews’ scores, NSF’s merit review process 
incorporates peer review in a system that also considers those attributes of a pro- 
posal (risk, multidisciplinary nature, novelty) that are not readily accommodated by 
a numerical score, but essential to identifying the most innovative proposals.” 

The National Academy of Sciences, in the 1994 report “Major Award Decision- 
making at the National Science Foundation,” stated that, “The United States has 
built the most successful research system in the world. The use of peer review to 
identify the best ideas for support has been a major ingredient in this success. Peer 
review-based procedures such as those in use at NSF, the National Institutes of 
Health, and other federal research agencies remain the best procedures known for 
ensuring the technical excellence of research projects that receive public support.” 
In November 2009, the Executive Director of the Transportation Research Board at 
the National Research Council, provided testimony before Congress on how to facili- 
tate the implementation of research at the Department of Transportation. In that 
testimony, the Director endorsed strongly the fact that NSF’s merit review process 
is well suited to the mission of the agency. His observation: “The more applied miti- 
gation and adaptation research topics should be steered by the concerns and needs 
of policy makers and practitioners, while the fundamental research topics should be 
organized along the NSF model in which scholars and experts are guiding the deci- 
sions about which projects are likely to be most promising.” 

NSF’s merit review process has served the agency, the scientific community, and 
indeed the country well for many years. Many Nobel Laureates, National Medal of 
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Science and Technology winners, and MacArthur Foundation Fellows (popularly 
known as recipients of Genius Grants) have been supported by NSF at various 
stages in their careers. Through separate programs and in the course of funding 
specific scientific progress, over the past 25 years NSF has also supported the train- 
ing of hundreds of thousands of graduate and post-graduate scholars in STEM 
fields. Discoveries stemming from NSF-funded projects have led to advances across 
all areas of science, engineering and education, with far-reaching impacts in the 
fields of nanotechnology, information technology, environmental science, genomics, 
STEM education, and many others. 

The high quality of NSF’s merit review process is recognized globally, as evi- 
denced by the fact that it has been used as a model by countries around the world 
that are newly establishing their own funding agencies. The merit review system 
for L’Agence Nationale de la Recherche (ANR), the French counterpart to NSF, is 
explicitly modeled after NSF, as is that of the Foundation for Polish Science. NSF 
helped the European Research Council establish its merit review system some five 
years ago, and was instrumental in helping Ireland establish Science Foundation 
Ireland. Back in 1986, a Chinese official came to NSF for six months to learn about 
our merit review and decision making processes, and subsequently incorporated 
what he had learned in establishing the National Natural Science Foundation of 
China (NSF 09C). These are just a few examples of international agencies where 
NSF has had an explicit role in helping develop their merit review systems, but 
there are literally dozens of others that have borrowed our approach over the years. 

As the nature of research and the scientific enterprise continues to change — be- 
coming more interdisciplinary, technological, international and collaborative — NSF 
continues to explore ideas and strategies that could strengthen the merit review 
process by enlarging the range of tools that can be used in proposal evaluation. 
These ideas have come from a variety of sources — internally, from the research com- 
munity, from the practices of other funding agencies, and from the scientific lit- 
erature on merit review. One idea that we are actively exploring is a greater use 
of technology-mediated virtual panels when and where it makes sense, with the 
hope that decreasing the travel burden will expand the potential pool of reviewers. 
Among the benefits that NSF would derive from an expanded pool of reviewers are 
the inclusion of more and varied perspectives, increased opportunities for participa- 
tion by underrepresented groups, decreased review burden per individual reviewer, 
and decreased travel costs for the agency. We have established an internal working 
group to identify other viable candidates for pilot activities, and to develop plans 
for running and evaluating those pilot activities. We will be discussing these with 
an advisory committee over the next few months to get their help in refining the 
processes. 

For over 60 years, NSF has been forward looking in terms of how the agency man- 
ages its research and education portfolio. Merit review fosters the “process of dis- 
covery,” the means by which researchers can identify emerging scientific challenges 
and innovative approaches for addressing them. NSF is dedicated to ensuring that 
the merit review process remains robust, rigorous, and beyond reproach, in support 
of our mission and enabling us to pursue our goal of funding the world’s best re- 
search in science, engineering and education. 

I appreciate the opportunity to appear before the Subcommittee to speak to you 
on this important topic. I would be pleased to answer any questions that you may 
have. 

Appendix I: NSF Proposal and Award Process and Timeline 
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Chairman Brooks. Thank you, Dr. Marrett. 

I now recognize our second witness, Dr. Keith Yamamoto, for five 
minutes. 


STATEMENT OF DR. KEITH R. YAMAMOTO, 

VICE CHANCELLOR FOR RESEARCH, 

UNIVERSITY OF CALIFORNIA SAN FRANCISCO 

Dr. Yamamoto. Good morning. Chairman Brooks, Ranking Mem- 
ber Lipinski, distinguished Members of the Subcommittee. Thank 
you for the invitation to present a statement before you today. I am 
Keith Yamamoto, a molecular biologist, professor, and adminis- 
trator at the University of California San Francisco. My lab’s re- 
search has been funded continuously for 35 years by grants from 
the NIH and NSF. 

I have also been active in matters of science and public policy, 
leading or serving on dozens of committees focused on a broad 
range of these topics. One of these topics has been the federal merit 
review system for evaluation of biomedical and biological research 
grant applications, especially those overseen by NIH and to a lesser 
extent the NSF. 

Today, I shall describe the key conceptual and operational fea- 
tures of the merit review process, assess how well that process 
works from my point of view, consider the impact of a specific pro- 
posal to change the NSF process, and mention two modifications 
that might make merit review even stronger. 

Every application for NIH and NSF life sciences research sup- 
port, and at NIH, that means 80,000 applications per year, we just 
heard the 55,000 at NSF, every application is rigorously reviewed 
and prioritized for scientific merit by special communities of ex- 
perts. The details of the review process differ among the different 
federal agencies but there are two critical features that to my 
knowledge are common across agencies. 

The first is that the merit review is carried out by other sci- 
entists because only scientific peers have the knowledge and per- 
spective required to assess the scientific significance, innovation, 
and impact of proposed research projects, and the second is that 
the review committees judge only scientific merit. It is left to oth- 
ers to assess relevance of the applications to the goals or portfolio 
of the funding agencies and to make the funding decisions them- 
selves. Thus, peer review with a solitary focus on scientific merit. 

How does the process work and how well does it work? The re- 
view committees are federally chartered and populated by volun- 
teer expert scientists who set aside time from their own research 
at institutions across the country to carefully evaluate written pro- 
posals for scientific investigation. The reviewers exercise their indi- 
vidual and collective scientific judgments, prioritizing the scientific 
merit of the applications. To help avoid nonscientific biases, both 
the applications and the reviews are standardized. For example, in 
the NIH process, every application is reviewed and scored accord- 
ing to the same set of criteria, significance, approach, investigator, 
environment and overall impact. By any measure, peer-driven 
merit review has been spectacularly successful at identifying and 
prioritizing the most interesting, innovative and significant sci- 
entific research, thus enhancing profoundly the strength of feder- 
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ally funded research. In contrast, merit review is not intended to 
influence the breadth or type of research that is funded. Rather, 
breadth and type are affected by funding mechanisms and by the 
stated missions of the funding agencies. 

How might the merit review process be changed to further 
strengthen it? Let me mention first a change currently under con- 
sideration that would, in my view, not serve the process and with 
two examples of changes that might enhance it. 

The National Science Board Taskforce on Merit Review has pro- 
posed a revision of the broader impacts criteria for NSF merit re- 
view, and I am quoting, “to ensure consideration of how the pro- 
posed project advances national goals.” This criterion departs, in 
my view, from the singular focus on scientific merit that is essen- 
tial to the merit review process and calls upon reviewers to judge 
grant applications by metrics outside of their expertise. 

Moreover, NSF is the Nation’s basic research agency as man- 
dated by the NSF Act of 1950, which created the Foundation. The 
impact of individual basic research projects on broad national goals 
may be impossible to judge at the time of proposal but may emerge 
with time in unexpected and profound ways. 

Broad national goals of course are vital and need to be advanced 
collectively by federal science research agencies. They cannot, how- 
ever, be mandated for individual NSF grant applications. Indeed, 
national goals can be addressed by development of funding mecha- 
nisms and by defining agency priorities, not by the merit review 
process. 

Now, while the current merit review process has great strengths, 
there are potential changes that might improve it further. I cite 
several in my written testimony. Let me mention just two here. 
Consider recognizing and validating that a grant application is not 
a contract for sure, in fact, not even a roadmap of experiments but 
rather an exercise in which the applicant demonstrates his or her 
capacity to pose a scientific problem and devise a research plan 
that would impact and advance a field. In reality, scientists pursue 
the implications of each day’s results rather than adhering to the 
course of experiments imagined in their grants. Thus, grant appli- 
cation formats and merit review criteria would closely assess the 
merits of a proposed idea and of the investigator while reducing the 
current focus on experimental detail and feasibility. 

Second, consider formally denoting two separate classes of re- 
search, one I will call innovative and one we have been referring 
to as transformative, both essential but each requiring distinct 
merit review mechanisms. Innovative research would advance and 
deepen our understanding of current paradigms. Transformative 
research would disrupt or destroy prevailing paradigms and force 
creation of new ones. 

In closing, the merit review process for federal support of re- 
search is indisputably the best system for identifying highest qual- 
ity of science. Its primary features, peer-driven review and singular 
focus on merit, have been critical in identifying grant applications 
that describe the best science by the best scientists. A healthy and 
robust merit review process is critical to maintaining and extend- 
ing U.S. leadership in scientific research and innovation, which in 
turn is essential to reaching broad national goals. 
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This concludes my testimony. I will be pleased to answer ques- 
tions and address your comments. Thank you again for the oppor- 
tunity to discuss this important matter with you. 

[The prepared statement of Dr. Yamamoto follows:] 

Prepared Statement of Dr. Keith R. Yamamoto, 

Vice Chancellor for Research, 

University of California San Francisco 

Good morning, Chairman Brooks, Ranking Member Lipinski, and Members of the 
Subcommittee. Thank you for the invitation to present a statement before you 
today. I am Keith R. Yamamoto, Vice Chancellor for Research, Executive Vice Dean 
of the School of Medicine and Professor of Cellular and Molecular Pharmacology at 
the University of California, San Francisco. I received a Bachelor of Science from 
Iowa State University and a Ph.D. from Princeton University before migrating to 
San Francisco, where I have been on the faculty for 35 years. My molecular biology 
lab has been studying the detailed mechanisms by which small molecules made in 
our bodies, hormones, control important physiological processes such as metabolism, 
stress responses and immunity; in the course of that work, I have had primary re- 
sponsibility for training approximately 100 graduate students and postdoctoral 
scholars. Our research has been funded throughout by grants from the NIH and 
NSF. 

For the past 30 years, I have also been active in matters of science and public 
policy, leading or serving on dozens of committees focused on a broad range of 
issues, challenges and opportunities. One of the major areas of emphasis for those 
activities has been federal merit review system for evaluation of biomedical and bio- 
logical research grant applications, especially those overseen by the National Insti- 
tutes of Health (NIH) and the National Science Foundation (NSF). For example, I 
co-chaired the NIH effort to assess and enhance its peer review process, and served 
on the National Science Board/NSF task force on transformative research. These ex- 
tensive experiences have provided me with a rather deep perspective on the merit 
review process and its relationship to the U.S. life science research enterprise. 

In my testimony today, I shall: (1) describe the key operational and organizational 
features of the merit review process for biomedical and biological research; (2) as- 
sess how well that process works; (3) consider the impact of a particular change to 
the process that is currently being considered; and (4) enumerate some potential 
modifications that might further improve the process. 

Every application for NIH and NSF life sciences research support (NIH, for exam- 
ple, receives some 80,000 applications per year) undergoes rigorous review and 
prioritization of scientific merit by committees of peers, typically by scientists who 
themselves hold grants from the same agency, and whose research is in the same 
area of research as the proposed study. The details of the review process differ 
among the different federal agencies, but evaluation by peers is the crucial common 
feature. 

To populate the federally chartered merit review committees, agencies enlist vol- 
unteer service from expert scientists in each area of research, who agree to set aside 
time from their own scientific research at academic and research institutions around 
the country, typically several times a year, to carefully evaluate written proposals 
for scientific investigation. The reviewers exercise their individual and collective sci- 
entific judgments free of other biases, prioritizing the scientific merit of the applica- 
tions. 

To help prevent nonscientific biases, the formats of both the applications and the 
reviews are tightly delineated, whereas the applicants have broad flexibility in 
choosing the scientific subject matter of the applications. In the NIH process, for ex- 
ample, a standardized set of five criteria (significance, approach, innovation, investi- 
gator, and environment) is mandated as the basis for rating every application. As- 
sessment of the investigator is focused solely on past performance and qualification 
for carr 3 ring out the proposed study. Importantly, merit review committees are 
charged to judge only scientific merit. In particular, they do not assess relevance 
of the applications to the portfolio of the funding agency, nor do they not make fund- 
ing decisions. This singular focus on merit (together with peer-driven review) is the 
second key feature of the merit review process. 

Thus, it is important that decisions of scientific merit are insulated and separated 
from decisions of funding. In addition, of course, the merit review process has no 
control over the level of funding allocated to support meritorious applications. When 
funding levels fall far below the capacity to support some of the very best applica- 
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tions (as is presently the case), the merit review process appears to fail, i.e., out- 
standing science goes unfunded. This apparent failure instead reflects a misalign- 
ment between the number of highly meritorious applications and the number of dol- 
lars available to fund those applications. 

Any merit review process that depends upon peers to carry out evaluations must 
acknowledge and address at least two intrinsic and related conflicts of interest: re- 
viewers might unfairly support applications from their friends to create an “old 
boys” network, or they might unfairly disadvantage applications from competitors 
or from those outside of some perceived “inner circle.” In general, these intrinsic 
conflicts have been addressed successfully by well-crafted regulations, and more im- 
portantly, by a universal “culture of respect” from the participating scientists who 
serve as reviewers. 

While peer-driven merit review plays a crucial role in identifying excellence 
among proposed ideas and research plans, it is also the case that an element of con- 
servatism is intrinsic to peer review, which complicates recognition and 
prioritization of “transformative” ideas and approaches. This is because the majority 
of scientists, and therefore the majority of peer reviewers, embrace and extend pre- 
vailing scientific paradigms, whereas transformative research disrupts or destroys 
accepted paradigms and creates new ones. Because both types of research are essen- 
tial, approaches to address intrinsic conservatism are important. One strategy is to 
adopt special funding mechanisms, and perhaps some special elements of the review 
process, designed to identify ideas. 

The current merit review processes, which have been in place in the U.S. for over 
65 years, recognize that only scientific peers have the knowledge and perspective 
required to assess the relevance, innovation and impact of proposed research 
projects. Indeed, the current system of peer-driven merit review is widely held to 
be not the “best good system” for evaluation and prioritization of merit, but “the 
only good system.” 

By any measure (e.g., quality of scientific publications resulting from support of 
meritorious applications, honors and prizes given in recognition of the highly meri- 
torious research, products and services that are developed from the results of sup- 
ported research, creation of an outstanding scientific workforce resulting from train- 
ing of students and fellows supported), peer-driven merit review has been spectacu- 
larly successful at identifying and prioritizing the most interesting, innovative and 
significant scientific research projects. Hence, the merit review process enhances 
profoundly the strength of the research funded by the federal government. In con- 
trast, the merit review process is not intended to influence the breadth or type of 
research that is funded. Rather, breadth and type are strongly influenced by fund- 
ing mechanisms, and by the range and diversity of disciplinary foci that are chosen 
by the funding agencies. 

Are there specific changes that might further strengthen the merit review process 
to ensure support of the best science? Let me mention first a change currently under 
consideration that would in my view damage the process, and then end with a brief 
enumeration of some potential changes that might enhance it. 

The National Science Board (NSB) Task Force on Merit Review is currently re- 
viewing the NSF’s merit review criteria, and has proposed a revision of both the “In- 
tellectual Merit” and the “Broader Impacts” requirements, with the goal of clarif 3 dng 
their intent and the ways that they would be used in the merit review process. The 
purpose of the Broader Impacts criterion “is to ensure the consideration of how the 
proposed project advances a national goal(s).” This criterion, as stated, would in my 
view adversely affect the merit review process because it departs from the singular 
focus on scientific merit that is essential to the process, and because it obligates 
peer reviewers to judge grant applications by metrics outside of their expertise. 

This being said, I concur fully that broader national goals are essential, and as 
stated in the NSB Merit Review Principles, “collectively, NSF projects should help 
to advance a broad set of national goals.” It is important, however, to remain mind- 
ful of the language of the NSF Act of 1950, which directs the Foundation “to initiate 
and support basic scientific research and programs to strengthen scientific research 
potential and science education at all levels.” Indeed, NSF itself originated from 
“Science the Endless Frontier,” the redoubtable 1945 policy initiative of Vannevar 
Bush, which called out untargeted basic research as “the pacemaker of technological 
progress” in which “new products and new processes are founded on new principles 
and new conceptions, which in turn are painstakingly developed by research in the 
purest realms of science.” 

This implies that broad national goals should be advanced by the composite feder- 
ally funded scientific research endeavor, and in particular should not be mandated 
for individual NSF research grant applications. In general, such goals should be ad- 
dressed by development of funding mechanisms and by defining agency priorities. 
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and not as a part of the merit review process. Moreover, it seems that broad na- 
tional goals might specifically be mandates for mission-driven agencies that seek to 
support research relevant to health, environment, energy, food and agriculture, or 
national security, rather than for the National Science Foundation. 

While the current merit review process has great strength, there are conceptual 
and operational aspects that might improve it further. Some examples for consider- 
ation across a wide spectrum: 

• Reconfigure the grant application conceptually to be viewed less as a “contract,” 
and more as demonstrations of an investigator’s capacity to identify an important 
scientific problem and devise tests that could advance knowledge and under- 
standing, and thereby impact the field. Thus, the grant application is not intended 
as a “roadmap of experiments” projected three to five years into the future. Grant 
application formats and merit review criteria should place greater focus on the mer- 
its of the proposed idea and of the investigator, whole reducing the current focus 
on proposed experimental details and feasibility. 

• Motivate top scientists to maintain active participation in the merit review 
process, in part by developing mechanisms that more effectively encourage appli- 
cants to submit bold scientific ideas. 

• Establish and formalize two separate investigator-initiated funding mecha- 
nisms, innovative and transformative, which invite, identify and support research 
that, respectively, advances and deepens our understanding of current paradigms, 
or disrupts and destroys prevailing paradigms, and forces creation of new ones. Con- 
sider unique aspects of merit review process for transformative applications. 

• The topical/disciplinary focus of review committees has been eroded by the de- 
mands of reviewing a rapidly increasing proportion of grant applications that in- 
clude a remarkable diversity of experimental approaches. To recover the conceptual 
focus of merit review committees, institute a “focused external review” process in 
which ad hoc reviewers are requested to contribute electronically brief comments 
that address solely those technologies or approaches for which specific expertise is 
lacking on the chartered committee. Such external reviewers would not be asked to 
assess the overall scientific merit of the application; that responsibility would reside 
solely with the chartered committee. 

The merit review process used for review of federal grant applications for support 
of biological and biomedical research is indisputably the best system for managing 
this important responsibility. Its primary features of peer-driven review and sin- 
gular focus on merit have been critical in identifying grant applications that de- 
scribe the best science by the best scientists. Once unique in the world, the process 
is being widely emulated. 

This concludes my testimony. I would be pleased to answer your questions or ad- 
dress your comments. Thank you again for the opportunity to discuss this important 
matter with you. 

Chairman Brooks. Thank you, Dr. Yamamoto. 

I recognize our next witness, Dr. Nancy Jackson, for her five 
minutes. 


STATEMENT OF DR. NANCY B. JACKSON, 
PRESIDENT, AMERICAN CHEMICAL SOCIETY 

Dr. Jackson. Good morning. Chairman Brooks and distinguished 
Members of the Subcommittee. 

As President of the American Chemical Society (ACS) during 
2011, the International Year of Chemistry, it is my great pleasure 
to address the Subcommittee this morning. Founded in 1876, ACS 
is the world’s largest scientific society with more than 163,000 
members. A nonprofit organization, ACS was chartered by the Con- 
gress in 1937 to advance chemistry in all its branches, promote sci- 
entific research and inquiry, and foster public welfare and edu- 
cation. ACS members work in industry, academia and government. 
ACS is a long-time supporter and strong supporter of NSF and its 
merit review process, which is recognized globally as the gold 
standard for identifying the best research to fund. My testimony 
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today will concentrate on chemistry and NSF’s impact on our 
science. 

The Federal Government is an important source of support, par- 
ticularly for basic research conducted within the chemical enter- 
prise that also trains tomorrow’s researchers. By stimulating inno- 
vation, the Federal Government empowers a competitive U.S. 
chemical enterprise, which contributes to U.S. economic growth. 

Mr. Chairman, I would like to focus on three areas from my writ- 
ten testimony that I believe are of particular value in these times 
of constrained resources and economic challenges. Those three 
areas are: one, balancing the NSF research portfolio; two, stream- 
lining the merit review process; and three, measuring return on 
NSF investment. 

My first point has to do with portfolio management. As anyone 
with a retirement fund has been told, managing a portfolio is crit- 
ical to its long-term strength, finding the right balance between 
proven performing stocks and riskier, higher-return investments. 
Managing a research portfolio is similar, striving to find the right 
balance between more incremental research that builds on previous 
successes and those ideas that are riskier and could lead to game- 
changing developments. In times of tight budgets and restricted 
funding, NSF mustn’t become too conservative. The agency must 
ensure adequate attention to providing opportunities for young re- 
searchers as well as for out-of-the-box research that may create 
economic renewal, produce jobs and train the U.S. scientific work- 
force of the future. 

Secondly, NSF should be able to triage the least competitive sub- 
mission from the review process. At the present time, all proposals 
submitted to NSF must go through an extensive peer review, even 
if they are intellectually weak with no hope of ultimately getting 
NSF funding. For every 100 grant proposals a program officer re- 
views, a small number, perhaps around 10, will be of such high 
quality that it is obvious to everyone that it should be funded. An- 
other 50 proposals easily will be recognized as not competitive. 
However, in the current system, they all take considerable time to 
review and process. The real agonizing choices must be made con- 
cerning the remaining 40 proposals, all very good proposals. Under 
current funding, only 13 of those 40 proposals would be funded. 
The merit review process should be focused on these choices. 

Over the last 10 years, the number of grant proposals submitted 
to the NSF chemistry division has doubled while the size of the 
NSF chemistry division staff has remained the same, thereby pre- 
senting a significant challenge to NSF to continue to perform its 
job of identifying the best science. 

ACS is responsible for a research fund, albeit small compared to 
NSF. It is the ACS Petroleum Research Fund (PRF). It has seen 
the number of grant proposals skyrocket in recent years as well. To 
help manage this increase, a new policy was recently implemented 
that empowers PRF research managers to reject proposals at the 
outset that they deem to be poor in quality. The practical result 
has been that more time is freed up for PRF staff and the volun- 
teer peer reviewers to focus on selecting the right balance of re- 
search from the strong proposals. Empowering NSF research man- 
agers to likewise would be a practical step to focus a steadily in- 
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creasing NSF workload and maintain excellence in its merit review 
process. 

And that brings me to my third point. We need new tools and 
methods to evaluate the success of NSF investment. While ACS 
strongly supports NSF’s broader impacts criteria, we need better 
tools to determine their overall effectiveness. Measuring how many 
fellowships or grants are funded is easy; measuring increased inno- 
vation, improved national security or broadening participation is 
complex and must take place over a longer time scale. 

It is difficult to single out how one individual effort has impacted 
a complex collection of national priorities. I understand that find- 
ing the right metrics for measuring these issues is very difficult but 
it is worthwhile. Empowering NSF to more easily gather data is 
necessary to ensure that research is successful in achieving the 
broader impacts and that the selection process is meeting its many 
goals. 

Chairman Brooks, I thank you for the opportunity to testify 
today and I would be happy to answer any questions you or your 
Subcommittee have. 

[The prepared statement of Dr. Jackson follows:] 

Prepared Statement of Dr. Nancy Jackson, 

President, American Chemical Society 

As president of the American Chemical Society, or ACS, it is my great pleasure 
to address the Subcommittee this morning on the topic of the merit review process 
of the National Science Foundation (NSF). 

Founded in 1876, ACS has grown to be the world’s largest scientific society with 
more than 163,000 members and one of the world’s leading sources of authoritative 
scientific information. A nonprofit organization, ACS was chartered by the U.S. Con- 
gress in 1937 to advance chemistry in all its branches, promote scientific research 
and inquiry, and foster public welfare and education. ACS members work in indus- 
try, universities and colleges, and at national laboratories. 

ACS is at the forefront of the evolving worldwide chemical enterprise. It is the 
premier professional home for chemists, chemical engineers and related profes- 
sionals around the world as well as a global leader in chemical information. ACS 
publishes 41 world-class scientific journals and operates the Chemical Abstracts 
Service, which provides the most comprehensive databases of disclosed research in 
chemistry and related sciences. 

Every year, ACS gives more than $11 million in grants for basic research in petro- 
leum and related fields through the Petroleum Research Fund (PRF). Twenty-five 
researchers, who were recipients of these grants, later went on to become Nobel 
Laureates. 

The Society also plays a leadership role in educating and communicating with 
public policy makers and the general public about the importance of chemistry in 
our lives. This includes identifying new solutions to global challenges, improving 
public health, protecting the environment, and contributing to the economy. 

ACS has been a strong, long-time supporter of the National Science Foundation, 
which is of particular importance at this critical time in our nation’s history. My 
testimony will concentrate on chemistry and NSF’s impact on our science. While I 
think my observations and recommendations are broadly applicable, chemistry is 
my area of expertise. 

Chemistry is the fundamental science that is at the heart of processes and prod- 
ucts that meet our most fundamental needs for food, shelter, and health, as well 
as developments and materials that are vital to advances in biotechnology, com- 
puting, and telecommunications. It is a keystone of U.S. manufacturing and is es- 
sential to a range of industries. 

America’s $720 billion chemical industry is one of our nation’s top exporters, with 
$171 billion in annual exports, which accounts for more than 10 cents of every dol- 
lar in total U.S. merchandise exports. Within the United States, the chemical indus- 
try employs 784,000 people and is a driver of innovation. The industry invests $55 
billion in research and development annually, and one in five U.S. patents is chem- 



28 


istry related. In addition, the industry contributes to human and environmental 
health. Drug innovations, made possible through chemistry, have helped increase 
life expectancy in the United States by 30 years over the past century. 

I mention this today because the success of the chemical enterprise is due largely 
to scientific and technological breakthroughs and advances made in industrial, aca- 
demic, and government laboratories. Although much of the nation’s chemical re- 
search is carried out by scientists, engineers, and technicians employed in industry 
and academia, the Federal Government is an important source of support, particu- 
larly for basic research conducted by our nation’s universities and government lab- 
oratories. By stimulating the roots of innovation, the Federal Government plays a 
fundamental role in ensuring the ability of the U.S. chemical industry to stay com- 
petitive in the long term. And because so many other industries depend on chemi- 
cals, the federal investment enhances the ability of the United States to compete 
globally by enabling a high-tech, competitive chemical industry to supply new prod- 
ucts at prices that give our nation’s producers an edge. 

The NSF plays a unique role in the U.S. scientific enterprise. While other federal 
agencies have missions directed at advancing specific science and technology in 
health or energy, for example, the core mission of NSF is to foster a healthy sci- 
entific enterprise here in America. Supporting the best ideas and exploring new 
frontiers across research disciplines have been the hallmark of NSF and the back- 
bone of the American research system. 

NSF has played a pivotal role in paving the way for scientific discovery, in large 
part, by awarding grants to members of the scientific research community that have 
demonstrated outstanding merit. The Foundation accomplishes its mission by sup- 
porting fundamental research and education in science and engineering. From air- 
craft design, pioneering medical tools and robotics, to discovering how children can 
learn chemistry better, NSF has played a key role in funding discoveries that have 
driven the nation’s economy, improved our quality of life, and enhanced national se- 
curity. It also supports high-risk research and novel collaborations that could deliver 
exceptionally high rewards. 

NSF is not just about research. It’s also about developing and training tomorrow’s 
scientific workforce. There is a symbiotic relationship between research and edu- 
cation. When a graduate student or a post-doctoral student works with a researcher 
funded by NSF, the student is honing skills and adding new scientific knowledge. 
In this way, the torch is passed from one generation of researchers to the next. To 
put it another way, this is how we keep pushing the edge of the envelope. If the 
United States is to continue to be a leader in science and technology, then we need 
to have the trained workforce working in that space. 

NSF provides more than 20 percent of the federal support for basic research at 
academic institutions and supports roughly 10,000 new awards per year through the 
merit reviews of over 40,000 proposals received. Every year, an estimated 200,000 
people, from undergraduates to senior faculty, participate directly in NSF research 
and education programs. 

The NSF merit review process is the gold standard worldwide, and is one of the 
reasons why U.S. science has been as successful as it is. When other countries seek 
to set up their own national research efforts, they often look to the U.S. NSF as 
the role model to emulate. 

At NSF, all proposals are evaluated for intellectual merit and broader impacts. 
NSF receives far more meritorious proposals than it could ever fund. While a pro- 
posal with weak intellectual merit has no hope of getting NSF funding, many pro- 
posals are rated “excellent” with strong intellectual merit and still do not get funded 
because of the stiff competition. The broader impacts criteria take into consideration 
which research is the most urgent or has the greatest relevance to improving the 
quality of life. This merit review process enables NSF to ensure that precious R&D 
money goes only to the most pressing R&D needs. 

As anyone with a retirement fund has been told, managing a portfolio is critical 
to its long-term strength. Financial advisors stress that it’s important to find the 
right balance between solid performing stocks and riskier investments that may pro- 
vide higher returns. Managing a research portfolio is similar: the research manager, 
whether working in industry or at NSF, strives to find the right balance between 
science that will deliver steady advances and ideas that are out of the box, but could 
result in game-changing developments. This point is especially important in times 
of restricted funding. It’s human nature to make more conservative choices and be 
risk-aversive when times are tough. However, now more than ever, America needs 
pioneering research that will create economic renewal, produce jobs, and train the 
scientific workforce of the future. Extra efforts and attention must be paid to culti- 
vating young researchers and game-changing ideas. 
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One of the reasons why the merit review process is so successful is because it 
draws from the collective wisdom of the scientific community. Many NSF personnel 
come directly from the scientific community and will return to their research insti- 
tutions at the end of their two- or-three-year rotations. Relying on rotating directors 
means the managers are up to date on the most recent scientific developments. The 
panels that perform the peer review of proposals are fellow researchers in the field, 
and as such, are also up to speed on the latest developments. This scientific commu- 
nity service, whether performed by grant proposal reviewers or NSF program offi- 
cers, is an integral part of scientific culture. Many scientists dedicate their time in 
this way because it provides an opportunity to remain in touch with and influence 
the cutting edge, as well as because they understand that the system only works 
if everyone volunteers to play their part. In a way, it is the science community’s 
way of llpaying it forward.” 

The merit review process requires significant efforts by both NSF employees and 
scientist volunteers. To better understand how the process plays out, consider this 
example from the NSF chemistry division. The division receives about 1,800 pro- 
posals annually. Each program officer in the division manages about 100 proposals 
a year. These managers are responsible for picking peer reviewers, and they must 
do so with an eye for diversity across a large number of factors such as ensuring 
that the reviewers reflect a balanced group based on type of institution (e.g., small 
undergraduate colleges vs. large research universities), geography, and racial and 
gender characteristics. 

Peer reviewers must also be experts within the proposal’s subfield of chemistry. 
Generally, a program officer approaches three reviewers to find one who will accept 
the call to serve. Since each proposal requires three to five reviewers, this means 
the officers approach six to 10 reviewers for each proposal. Therefore, on the aver- 
age, the NSF chemistry division approaches between 10,000 to 18,000 researchers 
to serve as peer reviewers for the proposals submitted. 

For every 100 grant proposals a program officer reviews, a small number (perhaps 
10) will be of such high quality that it is obvious they should be funded. Another 
50 proposals will be recognized as clearly not competitive; however, they still must 
be considered through the process. An agonizing choice must then be made over the 
40 proposals in the middle. These include proposals that may be considered excel- 
lent or very good. In fiscal year 2010, the NSF award rate was 23 percent. In our 
analogy of 100 proposals, this would mean that 13 out of the remaining 40 would 
be funded. 

The broader impacts criteria include considerations about whether the research 
proposal would broaden underrepresented minorities’ participation in science, 
strengthen U.S. infrastructure, improve national security, or foster innovation. 
Some of these impacts are the result of language in the America COMPETES bill 
enacted last year. The broader impacts criteria take into consideration which re- 
search is the most urgent or has the greatest relevance to improving the quality of 
life. The broader impacts criteria enables NSF to choose between meritorious and 
even more meritorious proposals, and is a way to ensure that precious R&D money 
goes to the most pressing R&D needs. 

It should be added that the number of grant proposals submitted to the NSF 
chemistry division steadily increases each year, more than doubling from levels 10 
years ago. And while the number of proposals has doubled, the size of the NSF 
chemistry division staff has remained the same. NSF is challenged to continue to 
perform its job of supporting the best science, even as the sheer number of proposals 
competing for funding has ballooned. 

I mentioned in my introduction that ACS is responsible for the management and 
administration of the ACS Petroleum Research Fund (PRF), which was established 
in 1944 by seven oil companies as a perpetual trust to advance science education 
and fundamental research in the petroleum field. In 2010, the Fund provided $11.4 
million for research grants. 

Although PRF is a small research fund, like the NSF, it has seen the number of 
submitted grant proposals sk 3 Tocket in recent years. Perhaps some insights gleaned 
from PRF would be useful in considering how to strengthen the NSF merit review 
process. 

Several years ago, to relieve the growing peer review burden on the science com- 
munity and to lighten the administrative load on the PRF staff, a policy was imple- 
mented to withdraw proposals from consideration that were deemed to be “poor” 
from the get-go: This includes those that are poorly written, use bad science, or do 
not address the specific scientific areas that were eligible for funding. In practice, 
this means that the managers now triage approximately 20 percent of the grants 
that come their way. These managers err on the side of caution: If there is any 
doubt that a proposal may have some merit, it is forwarded to the peer review pan- 
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els for consideration and ranking. The result has been that, while some “poor” pro- 
posals are removed from the evaluation process, more time and energy is freed up 
for PRF staff and the volunteer peer reviewers to focus on selecting the right bal- 
ance of research from the strongest proposals. 

Currently, NSF does not have the freedom to remove any proposals from the very 
bottom of the pile from consideration. Empowering NSF research managers to do 
so — provided that specific criteria are taken into consideration — would he a simple 
step to help NSF maintain excellence in its merit review process. If the average ac- 
ceptance rate for an NSF proposal is about 23 percent, this means that 77 percent 
of funding proposals will be turned away. Enabling managers to remove the lowest 
20 percent of those that would normally he rejected from consideration is highly un- 
likely to result in a potentially great proposal not getting its due consideration. In- 
stead, this approach may be a practical step to balancing a steadily increasing NSF 
workload. 

I have mentioned the important role the broader impacts criteria play in the NSR 
merit review process. NSF promotes broadening participation of underrepresented 
minorities and women, and persons with disabilities. This also includes increasing 
diversity in the NSF portfolio with respect to types of institutions supported and the 
geographic regions represented. Broadening participation is one way to address the 
broader impacts criteria; however, other activities are also appropriate. 

The importance of NSF efforts to broaden participation of underrepresented mi- 
norities in science and engineering is well understood and supported by the sci- 
entific community. We know that if the scientific work force doesn’t reflect the de- 
mographics of our country, we risk missing out on bringing the best minds and tal- 
ents from every community to work on the scientific challenges that will impact all 
of our lives. 

In spite of NSF efforts in the broader impact criteria areas, we could use better 
tools to measure how effective these NSF efforts have been. One difficulty of meas- 
uring the long-term impact of the broader impact criteria is that it’s easier to meas- 
ure the inputs than the outputs. Measuring how many fellowships or grants are 
funded is easy. Measuring increased national innovation, improved national secu- 
rity, or broadening participating is complex and must take place over a longer time 
scale. These differences make it difficult to single out how one individual effort has 
impacted a complex collection of national priorities. 

As a scientist, I want these efforts to be successful. As a senior administrator, I 
recognize that it’s nearly impossible to measure success if you can’t measure long- 
term outputs. In industry, we understand that finding the right metrics is very dif- 
ficult, but it’s worthwhile to try. I do not know how to resolve this issue, but I do 
believe that empowering NSF to more easily gather the data needed to measure the 
success of the broader impacts criteria would be a necessary step to ensuring those 
efforts achieve their desired effect to the maximum extent possible. 

Chairman Brooks, I thank you for the opportunity to testify today and to share 
these thoughts with you. ACS believes the NSF is the cornerstone of the U.S. sci- 
entific enterprise, and we stand ready to assist you efforts to strengthen the agency 
for the benefit of the scientific community and the entire nation. 

Chairman Brooks. Thank you, Dr. Jackson. 

Next, we recognize our final witness. Dr. Jose. 

STATEMENT OF DR. JORGE JOSE, 

VICE PRESIDENT FOR RESEARCH, INDIANA UNIVERSITY 

Dr. Jose. Good morning. Chairman Brooks, Ranking Member Li- 
pinski and distinguished Members of the Subcommittee, and espe- 
cially Representative Bucshon from Indiana. Thank you for allow- 
ing me the opportunity to speak with you today. 

My remarks arise from my experience as a primary investigator 
funded for many years by the National Science Foundation, as a 
member of NSF review panels, and an advisor for strategic plan- 
ning at both the NSF and the National Institute of Health (NIH). 
I have been the Vice President for Research at two major research 
universities, for five years at the State University of New York in 
Buffalo, and since last August at Indiana University. Both institu- 
tions are members of the American Association for Universities and 
the Association for Public Land-Grant Universities. 
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There are three points I would like to emphasize this morning. 
First, merit review is the most effective process we have for ensur- 
ing that federal funds are used to support the most important and 
far-reaching scientific research. Second, merit review is the best 
way to ensure impartiality in awarding funding to research. And 
finally, federal funding for scientific research awarded through rig- 
orous process of merit review is a necessary and important compo- 
nent of continued American preeminence in the world and is the 
foundation of our Nation’s long-term economic and national secu- 
rity. 

A very large number of scientific problems are interesting in 
principle but only a small number of those problems is important 
and deserves the investment necessary to search for and find solu- 
tions. I support what Dr. Jackson said about not all of the prob- 
lems that people want to study are interesting or relevant for 
study. My broad experience in the process by which the Federal 
Government funds scientific research at universities leaves me to 
conclude while no review system is perfect, merit review is the best 
way to identify those problems that are important to support. 

One indication of the strength of the merit review system is the 
number of major scientific breakthroughs resulting from research 
that was awarded federal funding as a result of peer review by ex- 
perts, people uniquely able to recognize the potential of proposed 
research. For instance, one can imagine how the average person in 
the street would respond if asked whether or not we should spend 
taxpayer dollars to study C. elegans, which is a tiny, transparent 
worm, a millimeter in size. Yet federally funding both at the NSF 
and the NIH on this worm has enabled scientists to identify key 
genes regulating organ development and programmed cell death 
and has shown that corresponding genes exist in high-end species 
including humans, of course. Nine recipients of Nobel Prizes since 
2002 focused their work on C. elegans. Research on C. elegans has 
led to clinical trials on the treatment of macular degeneration, 
asthma, diabetes and brain diseases. These advances occurred be- 
cause experts in the field recognized that this tiny worm would 
make a good model system to study and could help us learn impor- 
tant things about humans. 

I would like to comment on the broader impact criteria that are 
part of the merit review. In my experience, the broader impacts cri- 
teria are used only to decide between proposals of equal scientific 
merit, and this is entirely appropriate. Having said that, I would 
like to stress the importance of expanding the participation of 
underrepresented groups in the sciences, a goal which over the past 
10 years has been the primary broader impact criteria for NSF. 
This is not a matter of establishing quotas or prioritizing this par- 
ticipation over the scientific importance of a proposal but the NSF 
emphasis on this is the demonstrated way of increasing the pool of 
talented and trained scientists which in turn enhances American 
economic competitiveness and leadership in scientific research, dis- 
covery and innovation. 

In these challenging times, it is more important than ever that 
federal funds are spent wisely. Federal research funding has for 70 
years been the cornerstone of American economic security, sci- 
entific and educational preeminence, maintaining our competitive- 
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ness globally, continuing American leadership in scientific research 
and innovation and ensuring that our children and grandchildren 
enjoy a future in which American higher education and industry 
remain the envy of the world. All these rely on continued robust 
federal funding for scientific research and discovery, funding 
awarded through the merit review system as my colleagues have 
already stated. 

Thank you very much for inviting me here this morning and I 
look forward to answering any questions you may have. Thank you. 
[The prepared statement of Mr. Jose follows:] 

Prepared Statement of Dr. Jorge Jose, 

Vice President for Research, Indiana University 

Chairman Brooks, Ranking Member Lipinski and distinguished Members of the 
Subcommittee, it is an honor to be here this morning to speak with you about the 
important topic of the merit review process in federal funding for scientific research, 
and in particular at the National Science Foundation. My remarks today arise from 
my experience as a primary investigator funded for many years by the NSF, as a 
member of NSF review panels, and as an advisor for strategic planning at both the 
National Science Foundation and the National Institutes of Health. I also speak 
from my experience as Vice President for Research at two major research univer- 
sities, for five years at the State University of New York at Buffalo, and since last 
August, at Indiana University. Both institutions are members of the American Asso- 
ciation of Universities and the Association of Public and Land-Grant Universities. 
This morning I would like to speak with you about the importance of the merit re- 
view process, specifically, its integral place in establishing American preeminence 
in higher education, scientific investigation, and economic innovation. 

My broad experience in different aspects of the process by which the federal gov- 
ernment funds scientific research at universities leads me to conclude that while no 
system of review is perfect, nor guaranteed to fund only the best scientific research, 
the merit review system is the most effective process we have for ensuring that federal 
funds are used most effectively in support of scientific research, in particular at this 
time of limited resources when we need to prioritize how the teixpayer dollars are 
best invested. 

In some respects, the challenges facing federal funding agencies such as the NSF 
and the National Institutes of Health are very much like the challenges I face as 
Vice President for Research at Indiana University. As you may know, research uni- 
versities often invest some of their limited resources to catalyze programs of re- 
search into issues that are of fundamental importance to our State, our Nation, and 
the world. My goal is to help our researchers identify and address the most impor- 
tant scientific, social and economic problems ofthe 21st century, such as energy se- 
curity, health care, national security and our global competitiveness. Addressing 
these problems is not only valuable but is also a necessity because the problems will 
not solve themselves. Seemingly intractable problems can be solved only when the 
best minds with the appropriate expertise are brought together, and America’s cur- 
rent and future well-being depends in an essential way upon the results of research 
into these problems. 

As important as this work is, research is just one among many important areas 
of the University’s work and available resources — at lU as within the Federal Gov- 
ernment — are limited. Given limited resources and given the importance of tbe prob- 
lems, it is crucial that lU directs available resources to the projects with the strong- 
est likelihood of being transformative and successful. I rely heavily on the ability 
of experts to assess each proposal, the work plan, and the potential of specific people 
to carry out a project successfully. This guidance helps to ensure that funding deci- 
sions are made on the basis of scientific merit rather than personal or political con- 
siderations. In short, merit review must be the foundation of funding decisions we 
make at lU because merit review enhances the likelihood that we will properly in- 
vest our limited available funds into research projects with the strongest potential 
for innovation, for transforming a field, or addressing successfully an important 
problem. 

We are, of course, following the path that was set up by Vannevar Bush in his 
“Science — the endless frontier” developed right after World War II, that led to the 
formation of the National Science Foundation. NSF introduced the merit review 
process as an essential component to assess and determine how tax dollars should 
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be best allocated to scientific research. The merit review process has for the last 60 
years led to many NSF notable successes. It is very important to recognize that a 
very large number of scientific problems are interesting in principle but a much 
smaller subset of those problems is important and deserves the investment nec- 
essary to search and find solutions. 

Merit review is the best way to identify the important problems. It is the best 
way to ensure that federal funds are invested in a healthy array of important prob- 
lems, covering a breadth of areas and approaches within a particular field. And 
merit review is the best way to ensure impartiality, so that the best science and 
the best scientists are funded. 

In his planning for the NSF, Vannevar Bush drew on the experience of wartime 
scientific research, organized through the National Defense Research Council 
(NDRC). Wartime federal investment in scientific research resulted in the develop- 
ment of penicillin, the radar, and most famously, the atomic bomb. NDRC research 
brought together the very best scientific minds from Europe and the U.S. to work 
on the most important problems of the time — and resulted in discoveries that helped 
America become the pre-eminent economic, military, and scientific power of the 20th 
century. Since the end of the Second World War, federal investment in research at 
American universities has been central to the development of universities that are 
the envy of the world, and merit review has been central to funding research that 
enables the United States to remain the leader in scientific inquiry, the develop- 
ment of new technologies, and the translation of fundamental research into applica- 
tions that shape our lives every day. While there are many ways to assess and dem- 
onstrate the value of the merit review system, allow me to focus on one. The 
strength of relying upon merit review to determine what small percentage of pro- 
posals will be funded is demonstrated by its adoption by other countries across the 
world. Funding agencies in Europe, South America, and Asia, which are trying to 
emulate the research breakthroughs the U.S. has had in the last 60 years, all rely 
on a merit review system in which experts assess proposed research much as we 
do in the United States. 

I can give you a long list of items we use every day which were often developed 
as a result of the research that was funded by the federal investments in scientific 
research. One which the large majority of Americans use is the cell phone; another 
is the Global Positioning System, which was developed by the Department of De- 
fense but which was based on a trail of research discoveries that started with the 
work by Einstein on his mathematical theory of gravity. GPS use is so common that 
it is hard to imagine how we found our way anywhere before it became commer- 
cially viable! We also hear the weather report, in particular during these hot days; 
the food we eat has been produced using scientific breeding techniques. We often 
don’t think about where all these things came from, but we do know that Americans 
as a whole are very proud of the long tradition of scientific research, inventiveness 
and innovation, a tradition which has made the U.S. the advanced technological so- 
ciety that it is today. 

It might be helpful to reflect for a moment on a couple of specific examples of the 
impact of federal funding for scientific research, because these examples point to the 
success of merit review. They also demonstrate that it is not always possible to an- 
ticipate what kind of impact research may have. For example, Caenorhabditis 
elegans (c. elegans), a transparent worm that most of us can go our whole lives with- 
out thinking too much about, but which has been the subject of significant research 
since the early 1960s. Only one millimeter in length, it was the C. elegans which 
first had its full genome sequenced, prior to the big achievement of decoding the 
human genome. For many reasons, C. elegans is a useful model organism, enabling 
researchers to learn about genetics, cell biology, and the pathogenesis that relate 
to many human diseases. Three times since 2002, Nobel Prizes have been awarded 
to researchers working with C. elegans. The 2002 Nobel Prize in Physiology or Medi- 
cine was shared by three researchers, Sidney Brenner (Berkeley), John Sulston 
(Cambridge), and Robert Horvitz (MIT), whose work was funded largely by the Na- 
tional Institutes of Health through its processes of merit review. The 2002 Laure- 
ates identified key genes regulating organ development and programmed cell death 
and showed that corresponding genes exist in higher species, including man. The 
2006 Nobel Prize in Physiology or Medicine was awarded to longtime NIH grantees 
Andrew Fire (Stanford) and Craig C. Mello (U. Massachusetts) for their discovery 
of RNA interference in C. elegans — work which has led to clinical trials in the treat- 
ment of macular degeneration, asthma, diabetes, and brain diseases. Further, the 
2008 Nobel Prize in Chemistry was awarded to Martin Chalfie (Columbia) for his 
work on green fluorescent protein in C. elegans (along with Roger Tsien and Osamu 
Shimomura, who studied these proteins in other contexts). I still read many C. 
elegans papers in the current scientific literature that keeps unraveling important 



34 


new discoveries with very likely applications including useful drugs to treat dis- 
eases. 

You can imagine the response if you were to ask a regular person in the street 
if they felt that investing tax dollars would be justified to study a simple, almost 
insignificant worm. As the results show, this research has been a good investment — 
and it’s an investment that relied on the experts who recognized that this organism 
was indeed an animal model system which had many properties in common with 
higher organisms like humans and that it was worth stud 3 dng. It is important to 
note that a significant percentage of the Nobel Prize winners for the last 50 years 
have been American or working in the U.S. This can be connected with the exist- 
ence of our merit review system, which tries to fund and identify only the best and 
more promising ideas for funding. 

In the social and behavioral sciences, federal funding awarded through merit re- 
view has been at the foundation of substantial and important research. Elinor 
Ostrom (Indiana University) ^ and Oliver Williamson (Berkeley) won the Nobel 
Prize for their work on “Economic Governance.” Ostrom explored how communities 
often govern shared property and common resources more effectively than institu- 
tions do. Her research not only challenges the logical assumption about the ineffi- 
ciency of informal groups, but also demonstrates that economic analysis can help 
understand myriad forms of social organization. Williamson provided a theory of 
why some economic transactions take place within firms and other similar trans- 
actions take place between firms in the marketplace. Their work informs us about 
how to handle one ofthe most basic choices in human organization: When should de- 
cision power be controlled inside an organization, and when should decisions be left 
to the market or governments themselves. Ostrom is the only woman who has won 
the Nohel Prize in Economics, and she benefited from long-term funding from the 
Social and Behavioral Sciences section ofthe National Science Foundation. Those 
funds were awarded through merit review. 

Why is the merit review process for awarding federal funds considered 
a strong or beneficial process? What is the impact of the merit review proc- 
ess on the breadth, type, and strength of research funded by the Federal 
Government? 

At the center of the merit review process is peer review, in which experts review 
each proposal for its importance, soundness, and possible transformational and 
broader impacts when considering if it should be funded or not. The experts who 
serve on review panels not only have expertise in specific research; they also have 
an understanding of the broader context of the discipline. They are therefore 
uniquely qualified to assess whether an idea is important as well as interesting; the 
extent to which the research methods proposed and the qualification of the research- 
ers is appropriate for the problem under consideration; the degree to which a pro- 
posed research project has transformative potential. The strength of the merit re- 
view process rests largely on the service of subject experts, their willingness to read 
proposals carefully, often for free, so as to identify those most deserving of funding. 
It is important to recognize that the expected results from the proposed projects for 
funding are not known in advance; otherwise they would not be called research. It 
is for this reason that expert reviewers are able to best assess the probability of suc- 
cess of a project based on the previous track record of the investigators submitting 
the proposal and the track record of the reviewers on the subject matter under con- 
sideration. 

Of course, even the most well-intentioned group of subject experts is not immune 
from the biases, limitations, and agendas that are part and parcel of being human. 
However, in my experience as a reviewer for many years and after having submitted 
many proposals to the funding agencies, it is clear that the agencies try to reduce 
as much as they can all possible types of bias. By and large, most of the reviews 
by the experts are more than likely to arrive at the correct overall decision about 
whether or not it is appropriate to fund a proposal. There can be outliers to the gen- 
eral process, but reviewers for the most part are very strict about basing their deci- 
sions on scientific merit, which is why the percentage of proposals which are strong- 
ly recommended for funding is so low. 

At its best, the merit review process is a system of checks and balances not unlike 
our system of government, in that program officers can also offer a counterbalance 
to the limitations of the peer reviewers. Program officers bring to the decision-mak- 
ing process an awareness of agency priorities and funding trends that individual re- 


1 Distinguished Professor of Political Science, and Senior Research Director, Workshop in Po- 
litical Theory and Policy Analysis, Indiana University; Research Professor and Founding Direc- 
tor, Center for the Study of Institutional Diversity, Arizona State University. 
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viewers or review panels will not have. In selecting reviewers, framing the review 
process, and interpreting the panel’s recommendations, program officers place fund- 
ing decisions in a broader context than just subject expertise. Together, program of- 
ficers and reviewers have been remarkably successful at identifying the best pro- 
posals across a wide variety of fields, basing their funding decisions first and fore- 
most on the intellectual merit as assessed by the experts, the transformative poten- 
tial, and intrinsic importance ofthe research proposed. 

With limited federal resources, what role does the merit review process 
play in ensuring that the best scientific and potentially transformative 
ideas receive funding? How do the broader impacts criteria requirements, 
in addition to intellectual merit, affect these funding decisions? 

The merit review process is the best way to ensure that limited resources are di- 
rected to the best ideas. As I stated above, the collaboration of subject experts and 
program officers enhances the likelihood that funded projects will be the ones ad- 
dressing important questions, in ways that are methodologically appropriate and 
that can lead to transformative changes in the subject matter at hand. The merit 
review process is, in my view, the best way to minimize the potential for politicizing 
scientific research, and to ensure that limited funds are allocated as well as pos- 
sible, in particular when the federal financial situation is as precarious as it is at 
the moment. 

The broader impacts considerations offer important additional criteria for funding. 
Allow me to stress that in my experience, the broader impacts criteria are used to 
decide between proposals of equal scientific merit — the contribution of a proposed 
project to achieving the extra-scientific goals included within these criteria does not 
outweigh scientific considerations. That is, assuming two proposals offer programs 
that are of equal importance and potential, then the broader impacts criteria should 
be used to distinguish them. In my view, funding decisions made in this way find 
the appropriate balance between intellectual merit and broader impacts of research. 

As members of the Subcommittee are likely aware, in response to the America 
COMPETES Act of 2010, the National Science Board recently proposed an expanded 
list of broader impact criteria for NSF proposals, a list that does give some cause 
for concern. I do not believe that this expanded list might alter the appropriate bal- 
ance between intellectual merit and broader impacts in funding decisions. Rather, 
my concern is that the expanded list will diminish the National Science Founda- 
tion’s admirable and necessary leadership in promoting the participation of under- 
represented groups (women, racial, and ethnic minorities) in the sciences. Expand- 
ing the participation of underrepresented groups is an economic and intellectual ne- 
cessity. Science and scientific innovation are increasingly important to a strong 
economy, and so American economic security as well as American pre-eminence in 
scientific inquiry and higher education depend upon expanding scientific education, 
at both the K 0912 and higher education levels. To the extent that the expanded 
list of broader impacts criteria diverts attention from this priority, it risks damaging 
our economic and national security. ^ 

Expanding the participation of members of underrepresented groups in the 
sciences is not a matter of establishing quotas or prioritizing this participation over 
the scientific importance of a proposal. The NSF emphasis upon this is a dem- 
onstrated way of increasing the pool of talented and trained scientists. This in turn 
enhances overall American competitiveness, and over the long run, it will ensure 
that the United States continues to be the world leader in scientific research, dis- 
covery, and innovation. 

How does the merit review process work to ensure review impartiality 
for all applicants while maintaining high standards of excellence? 

Among the strengths of the merit review process is a degree of flexibility and 
breadth that other possible review systems would be unable to match. By this I 
refer to the ways in which our proposal review processes recognize that merit has 
many facets. For instance, an investigator at an early stage in her/his career cannot 
be expected to have the track record that a more senior investigator has — and yet, 
the early career proposal may be as meritorious and promising as the ones sub- 
mitted by the senior investigator earlier in their career. The NSF and other agencies 
have worked to develop programs that attend to differences in seniority, to allow 
the junior investigator to develop and flourish since they represent our future. CA- 
REER Awards, available through the NSF, are targeted specifically at researchers 
early in their academic careers. But outside of these programs, the merit review sys- 
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tern contributes to impartiality of reviews because it brings a group of scientific ex- 
perts together to review proposals. Barring personalities or idiosyncratic agendas 
that subvert the process, merit review increases the likelihood of fair and open re- 
view of all proposals over any alternative review process I am aware of. 

Please discuss any potentially novel ideas that should he considered in 
order to strengthen the process. 

I am, as you have likely gathered, a strong proponent of the merit review system. 
1 believe it is the process more likely to ensure that limited federal funds for re- 
search are awarded to the most significant scientific research, with the highest like- 
lihood of long-term impact. Nonetheless, the process is not perfect. 1 have alluded 
to the possibility that review panels are not free from bias that can influence their 
funding recommendations. One weakness of the peer review process is that sci- 
entists can be somewhat conservative. By this, I mean that it is easier to identify 
and fund proposals with the likelihood of incremental scientific advances than it is 
to identify and fund proposals with the potential for transformative breakthroughs. 
To the extent that bias toward the incremental means that we fail to fund innova- 
tive, paradigm-shifting research, we may be missing opportunities for precisely the 
kind of scientific advances that characterized federally funded research in World 
War 11. 

This conservatism is recognized by funding agencies, which have occasionally re- 
sponded by altering the charge given to review panels. “For example, when NIH 
concluded that it was not awarding enough high risk/high payoff grants, it changed 
its charge to panels accordingly.” ^ Adapting guidelines given to review panels is 
one way to ensure flexibility and strength within merit review processes. ARPA 
09E, the Department of Energy program that focuses on funding breakthrough re- 
search, offers another model that might strengthen the merit review process. By 
using a multi-part proposal and review process, ARPA 09E attempts to give review- 
ers a better understand of extraordinary proposals, and thus a better chance of rec- 
ommending the best and most feasible proposals for funding. Similar multipart 
proposal and review processes might effectively be adopted by other funding agen- 
cies, in specific programs if not throughout all funding programs. This might 
strengthen the merit review process by making funding of extraordinary projects 
with transformative potential increasingly likely. Of course, scientific research ad- 
vances slowly and having transformative discoveries is not as common as one would 
wish. Thus I am not advocating that funding be directed solely to risky research. 
Rather, I am suggesting that the overall research funding apparatus could he im- 
proved by a measured emphasis on potentially disruptive discoveries and having 
mechanisms that allow them to be funded. 

In these challenging economic times, it is more important than ever that federal 
funds are spent wisely. Continued federal funding for scientific research remains an 
important priority. Federal research funding — via the National Defense Research 
Council during the Second World War, and via the NSF and other government agen- 
cies since then — has for 70 years been a cornerstone of American economic security, 
scientific and educational preeminence. Maintaining our competitiveness globally, 
continuing American leadership in scientific research and innovation, and ensuring 
that our children and grandchildren enjoy a future in which American higher edu- 
cation and industry remain the envy of the world — all of these rely on continued, 
robust federal funding for scientific research and discovery. Yet each dollar we 
spend must be spent wisely — and a strong, flexible and rigorous merit review proc- 
ess is the best possible guarantee that American scientific research will continue to 
lead the way forward and that every taxpayer dollar is spent with the highest pos- 
sible return on investment. 

I thank the Subcommittee on Research and Science Education for allowing me to 
express my opinions about the merit review systems used by the federal agencies 
in general but in particular by the National Science Foundation, which funded a sig- 
nificant portion of my research for close to 25 years. I will gladly respond to any 
questions you might have. 

Chairman Brooks. Thank you, Dr. Jose, and thank you, panel, 
for the information that you shared with us. I am reminding the 
Members that Committee rules limit questioning to five minutes. 
However, if time permits and there are Members who wish to en- 


^ James Turner, “Best Practices in Merit Review: A Report to the U.S. Department of En- 
ergy,” APLU, December 2010, p. 7. www. aplu.org j document. docHd=2948. 

^ Ibid., p. 10. 



37 


gage in a second round of questions, we may do that. The Chair 
will at this point open the round of questions and as such I recog- 
nize myself. 

The first question is for Dr. Marrett. Based on fiscal year 2010 
data, please explain why over 2,700 proposals rated from poor to 
good to very good received federal funding over the 8,000-plus pro- 
posals that received very good to excellent or excellent ratings and 
were not funded. More particularly, in looking at the graph that we 
have been provided, there were some proposals, one that had a 
poor to fair rating but was funded, we had 98 proposals that re- 
ceived a fair to good rating that were funded, while on the other 
hand we had 1,312 proposals that received an excellent rating that 
were not funded, and similarly, 6,318 proposals that received a 
very good to excellent rating that were not funded. So generally 
speaking, why this kind of variation? 

Dr. Marrett. Yes. Now, I have to explain, to go back to one of 
my key points, the role of program officers who have to bring their 
expertise to bear. What you are reporting on are the results of the 
reviews of external panels. The external panels look at, as already 
indicated, very much the technical merit of proposals. There are 
other things that have to be taken into account, and they have 
been the founding of the Foundation. Thus, a program officer has 
to ask about what else might be known about the proposal. The 
program officer also pays a lot of attention to what Dr. Jackson 
mentioned, a portfolio. Thus, if we are to be concerned about how 
well we are serving the entire United States, how well a portfolio 
represents within a discipline the varying ways in which work 
might be done in that discipline so that there isn’t complete overlap 
in what is being funded. So making sure that there this is this kind 
of balanced portfolio is a responsibility that leads then to the fact 
that our program officers who make their recommendations to the 
next levels of review in the process, program officers are not then 
bound by the kinds of assessments that might be given since those 
assessments are on technical merit. They are not bound to have to 
rely exclusively on those recommendations from the external re- 
viewers. 

So let me close by saying, I will note again, those are rec- 
ommendations that are made and they are not the final decisions 
or how the funding will be done. 

Chairman Brooks. You mentioned other things taken into ac- 
count and then you gave as an example overlap. Are there any 
other factors taken into account that you have not shared? 

Dr. Marrett. Well, there can be a number of things. I think 
some of the questions from the Committee that we received asked 
about such questions as geographical balance. One of the questions 
commented on the fact that we know that talent is not limited to 
any specific part of the Nation and thus there is a question often 
of how are we looking across, but this is not to suggest that these 
are non-meritorious. You don’t start with the assumption that you 
will do away with the merit but the meritoriousness of proposals. 
So there are numbers of other things. The potential contributions 
to innovation can be another criterion that a program officer looks 
at. We then do have — and these don’t depart significantly from 
what will appear in all of the materials that we share with the 
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community, with what will appear in the broader impacts criteria, 
but all of these can be brought to bear as assessments are made 
about how to use the funding in the most effective ways for the Na- 
tion. 

Chairman Brooks. These other criteria that you mentioned, are 
they required by statute, required by Code of Federal Regulations 
or internal to the way in which the NSF makes its decisions? 

Dr. Marrett. They are internal to NSF. We do abide by the code 
that affects all the federal agencies when it comes to merit review 
so merit review in a broad sense. They are sets of principle shared 
by all federal agencies because of the code. 

But then there are the specific criteria approaches that the Foun- 
dation has outlined and those are the ones that we try to make 
sure — not try — we actually make sure they are communicated, un- 
derstood by the community, thus our Web sites, our communica- 
tions always are very explicit about what is going to be expected 
of those who submit proposals and how the process will work. 

Chairman Brooks. And finally, the graph that I have talks 
about these proposals in terms of numbers but not in terms of dol- 
lars, so I know no score a number, poor to fair, a number of pro- 
posals that fit that criteria, fair to good, good to very good, very 
good to excellent, and excellent and so forth. Does the NSF have 
this information broken down by dollar figures or is it strictly on 
a number basis? 

Dr. Marrett. We can provide you the information about dollars 
if you would like to know. 

Chairman Brooks. Will you please provide our Subcommittee 
staff with that information? 

Dr. Marrett. We will. 

Chairman Brooks. Thank you. 

Next I will recognize Mr. Lipinski for his questions. 

Mr. Lipinski. Thank you, Mr. Chairman. 

Dr. Marrett, as I mentioned in my opening statement, I offered 
a provision in last year’s COMPETES Act that authorized prize 
programs at all science agencies. This language began with a bill 
I introduced with Congressman Frank Wolf that would have cre- 
ated an innovation inducement prize pilot program at NSF. We in- 
troduced this legislation based on a 2007 National Academies re- 
port which concluded that “an ambitious program of innovation in- 
ducement prize contests will be a sound investment in strength- 
ening the infrastructure for U.S. innovation.” I have also worked 
with Congressman Wolf to include report language for the fiscal 
year 2012 Commerce, Justice, Science appropriations bill that asks 
NSF to make use of its new authority, especially the mechanism 
for funding high-risk, high-reward research projects. 

Dr. Marrett, can you tell me whether NSF has had any discus- 
sions on this topic and whether it has any plans to administer a 
prize program? 

Dr. Marrett. Thank you, Mr. Lipinski. We have had extensive 
discussions on the topic and appreciate the intent, or we appreciate 
what has happened to give us the authority to try out other kinds 
of tools in the toolkit. I appreciate your earlier comment that we 
must make sure that what we do fits into the kind of support for 
fundamental work that lies at the heart of what the National 
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Science Foundation does and so we have been looking at what 
would be appropriate in that context, and we will be launching a 
series of experimental efforts to try to see how these fit within our 
context and so we are — we haven’t been ready to announce very 
broadly because these are highly experimental but we are giving a 
lot of attention to the use of prizes and awards as ways for advanc- 
ing innovation and advancing knowledge in the Nation. So this 
would not be inconsistent with what our overall mission is but cer- 
tainly we have paid close attention to the strong interest being ex- 
pressed by this Subcommittee, members of the larger community in 
the fact that prizes and awards could be a mechanism for expand- 
ing our portfolio. 

Mr. Lipinski. Thank you. 

I want to turn now to Dr. Jose. One of the inherent challenges 
of NSF’s merit review process is overcoming potential conflict that 
exists when an established researcher reviews a transformative 
proposal that may question that reviewer’s work. Now, NSF pro- 
poses to address this challenge by awarding EAGER grants, re- 
training program officers and experimenting with alternative ap- 
proaches to the merit review process. In your testimony, you men- 
tioned a similar challenging experience by Indiana University and 
how crucial it is that lU directs available resources to projects with 
the strongest likelihood of being transformative and successful. 
Could you expand on how you ensure that funding decisions of 
transformative research are based on scientific merit rather than 
the personal considerations of reviewers, specifically, if such re- 
search conflicts with that of a reviewer? How does your process 
compare with NSE’s process, and if it differs, how do you rec- 
ommend incorporating Indiana University’s process into NSE’s 
process? 

Dr. Jose. Yes. Thank you very much for the question, Mr. Lipin- 
ski. One of the characteristics of a scientific endeavor is it moves 
slowly, usually moves slowly, and once in a while there is a big dis- 
covery and it has a big jump and then it continues. Scientists gen- 
erally actually are conservative. When they review proposals, they 
are conservative and they are less willing often to risk the funds 
that they have to decide to award to a researcher because they are 
not sure that it is going to succeed. Now, doing research means 
that we don’t know what the answer is going to be. That is why 
we call it research because we pose a question that we would like 
to find an answer. Often it is not easy to see how to get the answer. 
We have to have a plan of how we are going to do it. 

Now, when — we had a similar problem, as you mentioned, at In- 
diana University. We also have limited resources that we want to 
invest in trying to catalyze the research that the faculty may want 
to do that is risky and is transformative. Now, the way of doing 
that is by requiring from the reviewers to consider what will hap- 
pen if this project will succeed, how much is it going to change the 
nature of the field in which the proposal is being submitted, is it 
going to be transformative. What does it mean to be trans- 
formative? It is really not going to be just small change in the way 
things have been done but is really going to be significant in the 
way that it is being done. 
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The NSF and the NIH have recognized that that is a problem. 
In fact, I would say that the way they review proposals in the NSF 
is different than the way they review proposals in the NIH. NIH 
has review panels that are study sections that actually review pan- 
els for an extended period of time. In the National Science Founda- 
tion, the reviewers are usually confidential; their names are not 
known. And what happens is that the reviewers are often trying 
to make sure that the money is not going to be wasted so they are 
not as willing to risk funding a project that they are not so sure 
is going to work out. One needs to have some policies, and the NSF 
and NIH have done that. 

They have introduced projects, like the ones you have mentioned 
at the NSF, in which a certain amount of money is going to be 
given for projects that are risky but the payoff is going to be very 
high, if they succeed, it is going to be very high, money to be given 
for a project that actually going to be transformative. For example, 
the NIH has a program that is called Pioneer Awards. Pioneer 
Awards are given to projects that are very ambitious, very risky, 
that are really going to be transformative in the field in which they 
are given. So the agencies are trying to do that. It is not easy to 
do but it needs to be done. Having said that, funding proposals 
where the advance is slowly moving is also important because 
science doesn’t advance in jumps suddenly. Thank you. 

Mr. Lipinski. Thank you. Dr. Jose. Thank you, Mr. Chairman. 

Chairman Brooks. The Chair recognizes next Mr. Bucshon from 
Indiana. 

Mr. Bucshon. Thank you, and welcome. Dr. Jose. 

Dr. Jose. Thank you. 

Mr. Bucshon. A couple comments. First of all, I was a cardio- 
vascular surgeon in my previous career, and so I know a little bit 
about research and I would just like to say that a lot of discoveries, 
as everyone would recognize, are made by students and people who 
are just getting their careers started in science and so that is kind 
of the area that I am going to focus on a little bit. 

Some areas of research may only have a few experts in the field, 
so to speak, and the review process, even though technically exter- 
nal may actually be consistent with an in-the-family-type review, 
and that happens a lot, as we know, in science. So I guess. Dr. 
Marrett, I would like to ask, how do we guard against the politics 
involved? Because everybody knows there is politics involved in 
this process. In a highly competitive academic environment, both at 
the institutional level and at the individual level, and is there a re- 
view process for the reviewers that shows their consistency and 
their willingness to assess projects that appear to be nonpartial, 
even in this setting? 

Dr. Marrett. Let me note again first that what the reviewers, 
the panels that we call in or the mail reviewers, those are again 
advisory, and when you mention, we are very concerned to make 
sure that the process is free of lots of conflicts. There are a couple 
of ways in which those are handled. One is by having all reviewers 
do have to sign the conflict forms to indicate that they are not in 
conflict. Our program officers do training for ethics and conflict 
training. In fact, interestingly, we also have that now, ethics train- 
ing, for students, undergraduates, graduates, post-doctoral stu- 
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dents. There is also a requirement for institutions, that institutions 
must assure that there isn’t a conflict of interest that would he 
there. 

Even with all of that, we still have other sorts of processes to 
oversee how final awards are made and so the awards are not 
made exclusively on the basis of these recommendations. They go 
up through other levels for review. Then finally, if there still re- 
main problems, those can be undertaken through our Office of the 
Inspector General. If there is evidence of waste, fraud or abuse, 
those we would report to the Inspector General and the Inspector 
General gets them through other processes. So we have had — we 
put in place checks and balances to reduce the likelihood that there 
will be the kind of personal considerations that I know would al- 
ways be of concern, and that is why we have what I say again are 
the checks and balances to assist them that will not rely exclu- 
sively on any single individual. 

Mr. Bucshon. Thank you, because looking over the process, as 
you know, there is a disparity in the percentage of grants being 
made to new investigators versus established as well as univer- 
sities that traditionally get federal funding versus those that may 
be new to the process, and I understand all the reasons why those 
may occur. People honestly are more experienced at submitting 
their proposals and so they may be more likely to be successful. 
But the reality is, there probably is a little bit of this internal fam- 
ily-type thing that does come into play. Again, saying that, you 
know, students and new researchers are ones that frequently make 
the most discoveries. I am glad to hear your explanation that these 
people, you know, they appear to have the same opportunity. 

Dr. Jose, I will ask you a question. In your experience, many 
years of experience, has the review — I mean, is there a lot of polit- 
ical interplay in the review process, and you are free to 

Dr. Jose. I wouldn’t call it political. I think that often people 
that have been working in the field for some time, they think that 
they have the right answer, and when someone else comes in and 
say well, maybe I have another way of looking at the problem with 
a different answer, they may say oh, no, no, no. There is some re- 
sistance at the beginning. But eventually one of the nice things 
about science and the review process is that other researchers will 
actually come up with results that will say yes, the new way of 
looking at the problem is the way to do it. But it is always, just 
as in human nature, I mean, all of a sudden you have a way of un- 
derstanding a problem, you think you have the answer and some- 
one else comes in with a more innovative way of doing it and says 
oh, maybe not, maybe it is not a good idea to support it but eventu- 
ally I think the system works itself out to actually recognize new 
ways of doing things, and even when you have a group of people 
that are — what do you call them, in the family — that look at prob- 
lems in one way, eventually they have to give credit to the new 
ways of looking at a problem and solving the problem. 

Now, let me say something about the young investigators. The 
NSF has a program for — and NIH too — for recognizing junior inves- 
tigators that gives them some extra points, if you wish, when they 
start the running because they can’t compete with the ones that 
have been in the field for 40 years. So there are programs that are 
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working very well. They are not as good as they should he, hut they 
have recognized those problems. 

Mr. Bucshon. Great. Thank you. I yield back. 

Chairman Brooks. Thank you, Mr. Bucshon. 

The Chair next recognizes Mr. Clarke. 

Mr. Clarke. Thank you, Mr. Chair. We are a few days from pos- 
sible government default, and to me, I am the new guy here, rel- 
atively, like a few months here in Congress. Many years ago, I was 
a staffer. So I am going to say that maybe I have a perspective that 
could give us a clearer picture on things because I haven’t been 
part of the system for a long time. But one underlying issue that 
I see that Members of Congress have regarding taxpayer-funded 
basic research as we are dealing with here is they want to make 
sure that we are not wasting money, that it also goes to make a 
constructive difference in our country, and the broader impacts cri- 
teria kind of lays out what our objectives would be. 

Dr. Jackson indicated, I think, a very important issue, though, 
is that how do we best measure whether that research is effectively 
achieving those broader impact goals, especially when it is research 
that is totally innovative and totally transformative. It is almost 
like, you know, asking the veteran Members of Congress to quit 
blaming each other and think of a different way of actually passing 
legislation by inspiring the public with an enlightened point of 
view. I am not sure if that would actually come from entrenched 
folks here in this system but it could. 

So in that sense, I am just posing this question to all of you. 
What type of measures do you think we should look at? Dr. Jack- 
son indicated that maybe empowering the NSF to easily gather 
more data to measure the success of the research could be effective, 
but along those lines, we here in Congress want to make sure that 
when we are spending taxpayers’ dollars that the taxpayers are 
getting the best benefit for those dollars. How do we measure it? 

Dr. Jose. If I may answer the question, I mean, there is a pro- 
gram now, a federal program that is called NSTAR where they are 
going to try to find out for every dollar that is spent by the Federal 
Government, how many jobs it will create, how many people are 
educated, what is the impact that those dollars have in the econ- 
omy, for example, and the number of jobs and the competitiveness 
of the United States when comparing with other countries. It is a 
very difficult thing to do. It is a multi-variable type of problem. 
One answer is not that simple. We had this question asked of us 
when the stimulus money was given to universities, and we were 
asked, and I was interviewed a few times, and people would ask 
me, well, you are going to get $20 million so how many jobs will 
you create, how many jobs were created from that $20 million, and 
it was not an easy question to answer, particularly because the 
money had just arrived a few months before, and second, because 
in particular when it has to do with research, it takes some time 
actually to see the results. 

But I will say that in my view, the money that the United States 
has invested in funding research and particularly after the second 
World War has maintained the preeminence and the dominance of 
the United States economy. Furthermore, it has been the seed 
money that was used to create the best university system we have 
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in the world and is the envy of the rest of the world. I would call 
that evidence that everything has been working out. Now, having 
said that, we need to review constantly how we are doing things, 
how we are investing the money, because times change and we 
should look at things in a different way like today with the budget 
problem that we have, the Federal Government budget problem 
that we need to be very careful about how we invest money, federal 
research taxpayer money. 

Dr. Marrett. I would just elaborate a bit on that. I think the 
points are very well taken. The National Science Foundation pays 
attention constantly to issues of metrics because we take seriously 
our responsibilities to be accountable to the larger public. Those 
metrics are reflected in many ways in the kinds of requirements for 
what must be done, broader impacts. There are the assessments 
that have to take place as people are reporting on the broader im- 
pacts. So we are constantly looking at, reviewing, examining what 
would tend to be approaches that are possible. 

I think the other issue, a key issue here is, it is much easier 
often to think about short-term returns for those things that might 
have — we can look at, then the longer-term returns, and it is the 
longer-term returns that are very important for an agency such as 
the National Science Foundation. We have done studies that have 
looked at — we take account of some of important developments and 
track them back. Sometimes those have been over an extended pe- 
riod of time. Thus, I think a part of Dr. Jackson’s comments had 
to do with ensuring that our attention on returns would not focus 
us so heavily on immediate returns that we are unable to think 
about and invest in what will have to be the longer terms and the 
metrics appropriate for the longer life of activities. 

Dr. Jackson. And yes, I would say that is exactly true. When 
you analyze and measure the impact of fundamental and basic re- 
search, it takes a long time sometimes to see what those impacts 
are, so it is a long-term process, not just a short-term one. 

Dr. Yamamoto. I agree with that. I think you raise a really im- 
portant question, and the way that I would look at it, as I said in 
my remarks, is that this must be viewed as a continuum that rec- 
ognizes that application that really reaches a level of being evident 
in addressing national goals, requires a continuous input of funda- 
mental information about complicated processes and that for any 
one grant application, it could be impossible actually to ascribe how 
they may be approaching national goals. 

Let me do one quick example. My colleague. Herb Boyer, in the 
early 1970s was working on a process that was sort of an esoteric 
process in which DNA that got into bacteria was broken down for 
some reason and he didn’t understand, and this was called DNA 
restriction. The kind of DNA that would get into bacteria would be 
restricted by breaking it down. And very few people, very few sci- 
entists were interested in this process. I would wager that if this 
were put up against the test of addressing national goals, it might 
not do very well. That discovery of what DNA restriction modifica- 
tion was turned into the capacity to make recombinant DNA, which 
then turned into the biotechnology industry worth billions and bil- 
lions of dollars and leading the world. So it was impossible to as- 
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cribe with that particular grant what its impact would be but there 
it was. 

So I think that the goal would be in fact not to look at these 
things individually, to ask collectively whether the federally funded 
scientific research endeavor in this country is actually moving 
things toward national goals, to recognize that we understand so 
little about these processes that a continued input of investigation 
at the fundamental level is essential and that that continuum real- 
ly does work. 

The NSF is the basic research engine of the Federal Government 
and as such really needs to be celebrated and protected. There are 
mission-oriented agencies including the NIH in fact, the DOE, the 
DOD, Department of Agriculture, Commerce, elements of Com- 
merce and so forth that have specific missions and I think that 
those should be tested in the short term against whether they are 
addressing national goals but always to recognize that input of fun- 
damental science from the National Science Foundation and else- 
where is always going to be essential. 

Chairman Brooks. The Chair next recognizes Mr. Hultgren. 

Mr. Hultgren. Thank you, Mr. Chairman, and thank you all for 
being here. 

I have a question for Dr. Yamamoto and Dr. Jackson and Dr. 
Jose. Each of you has testified about broader impacts imposed on 
the Foundation by Congress in the 2010 American COMPETES Re- 
authorization Act and that it will adversely affect funding decisions 
by requiring reviewers to consider metrics far outside the scope of 
the intellectual merit. Dr. Jose suggests that rather than enhanc- 
ing competitiveness and national security, they risk damaging our 
economic and national security. I wondered, are reviewers even 
privy to the national priorities prior to the meeting of a panel, and 
just asking if you would comment, if each of you would comment 
on this. 

Dr. Jose. I would like to say that all of us are aware of our envi- 
ronment. All of us are aware about what is going on with Congress 
today. For example, all of us read the news, all of us know what 
is important. All of us are very aware about what is happening in 
the growth of the economies in the rest of the world and how the 
United States is having to compete very strongly with China, 
which in particular is growing very fast and we know or we feel 
that one of the ways in which the United States is going to succeed 
is by innovation, by being innovative, by being creative and intro- 
ducing new knowledge and new ideas, and I think that we should 
not limit the imagination of investigators and researchers by hav- 
ing a large number of restrictions about how the money is going 
to be awarded. I mean, we put too many restrictions that constrain 
the freedom that researchers have that often leads to the creation 
of an industry like biotechnology, as Dr. Yamamoto just said. It is 
the essence actually of the freedom of thinking freely that creates 
new innovations that were totally unexpected. 

Mr. Hultgren. Dr. Yamamoto or Dr. Jackson? 

Mr. Yamamoto. As I said in my comments, I think that one of 
the critical features of merit review is a singular focus on scientific 
merit and that anything outside of that is valuable and important 
and in fact essential as it may be such as addressing broader na- 
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tional goals should reside outside of that process and that asking 
merit reviewers to make an assessment of how well the fit is of a 
given project to national goals I think is inappropriate. I think the 
agency individually and collectively, as I said, the federal scientific 
research endeavor needs to he paying attention to whether national 
goals are being addressed but that neither reviewers nor appli- 
cants, in my opinion, should be mandated to make such a state- 
ment. If there is a clear application, of course, it would be to the 
advantage of the applicant to make that clear, but if not, and it is 
a fundamental question, then you are really asking reviewers to 
make guesses, and I think one of the great things about the merit 
review process is that there is not a lot of guesswork there and 
that we shouldn’t be asking reviewers to step outside of their ex- 
pertise. 

Dr. Jackson. As I mentioned in my testimony, there are a lot of 
proposals that are sent to NSF that are very, very good, excellent 
proposals that just cannot get funded because there is not enough 
money so that when the program officer at NSF looks at 40 pro- 
posals that are all great and could have a huge impact on this 
country and they have to decide which of those 40 to choose, then 
the broader impact does come as an advantage to choose between 
all these great proposals, and that is an important role, I think, for 
the broader impacts criteria and should be kept as such because 
there are clearly far more many proposals that are excellent that 
can be funded. 

Mr. Jose. May I say one more thing? 

Mr. Hultgren. Certainly. 

Mr. Jose. Just going along with what you said, I just came to 
Indiana University last year, and we have limited resources but we 
introduced a seed funding program for people that would be willing 
in interdisciplinary research. We actually asked people first to have 
new projects, new collaborations and so on and so forth. We know 
of about 165 proposals and we could only fund 10 percent. There 
were many that were recommended for funding but we had to 
make a decision at the end. We decided never to fund something 
that was not recommended on merit, but then we have to use some 
criteria to decide which projects were more likely to be good and 
producing new results or new discoveries and so we had to reduce 
the total number of projects that we were going to fund using other 
criteria. So it is important to have a combination perhaps of the 
two. 

Mr. Hultgren. My time is winding down. If I could just ask Dr. 
Marrett just to follow up quickly, Mr. Chairman? 

Chairman Brooks. We are going to do a second round, given our 
time availability. Go ahead and 

Mr. Hultgren. Let me just quickly. Dr. Marrett, I know this is 
something that Congress has directed you to do. It wasn’t nec- 
essarily something internal that you decided. I wonder if you would 
have any comment on what your colleagues at the table stated, 
what your feelings are about this. 

Dr. Marrett. Well, it is not quite true that this is only, that we 
only are talking about broader impacts because of Congressional 
activity. No, from the outset, from the time that NSF was founded 
and started making awards, it has always considered both the tech- 
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nical side and how we address the national goals. The idea, the ter- 
minology of broader impacts was really brought in in 1998 when 
the Foundation said we are now going to collapse several criteria 
down to the two criteria, intellectual merit and broader impacts. 
The broader impacts statement or the expectation is that the goals, 
these do have to reflect some of the kinds of goals that are expected 
by the Nation of the investments in science. So they would include 
economic competitiveness, development of a STEM workforce, in- 
creased participation by underrepresented groups, partnerships be- 
tween academia and industry. In other words, we have seen these 
as very critical for the Nation and reasonable as an agency thinks 
about how it will allocate resources. I think the only difference be- 
tween the NIH and NSF approach, NIH does much of the same 
kind of a thing, it is at a different level. So the advisory councils 
take up the questions of how should the portfolio and how should 
these other national considerations be taken into account. They 
happen to be done as a part of our regular merit review process 
in NSF. But this is not a new idea. 

Mr. Hultgren. But wouldn’t you say, though, it is an — this ex- 
pansion is mandated by COMPETES? Isn’t that true and that 
makes it unique? 

Dr. Marrett. No, what COMPETES did was to say be much 
clearer and there are other things with reference to broader im- 
pacts, and so that is what the Science Board is looking at, that is 
what the expectation of the Foundation, and that is what we do be- 
lieve we can be much clearer on what broader impacts should mean 
and how it would come up with indications appropriate metrics, 
but again, this was a part of an ongoing process and we appreciate 
then the interest of COMPETES, the interest of others in helping 
us continue to move forward in explaining and being accountable, 
again, to the Nation. 

Mr. Hultgren. Mr. Chairman, you have been very gracious. I 
yield back. Thank you. 

Chairman Brooks. Now for the second round of questions. The 
Chair will start first. 

Dr. Jose’s testimony stated that the merit review process is at 
best a system of checks and balances, and this question is for all 
of you. Being a member of a government body built to provide a 
system of checks and balances, I am interested in this theory. In 
the context of the financial circumstances that Congressman 
Clarke so notably described that we face here in Washin^on, DC, 
do you or any of your colleagues sitting with you today have any 
additional suggestions on how to strengthen the system based on 
our need for a balanced approach to federal funding, particularly 
to ensure the advancement of science as our sole or primary fund- 
ing criteria? 

Dr. Jose. Actually, let me actually say something. I mean, NSF 
has no specific mandate to do other than basic research and try to 
uncover new laws of biology or physics. The NIH has a very clear 
mandate to try to find the best way to improve American citizens’ 
health. The Department of Energy has very special goals, the De- 
partment of Defense, each one of them has a merit review system 
that works differently than the one at the NSF. The NSF is to 
some extent unique. 



47 


Having said that, I think that when I said checks and balances, 
what I mean is that the reviewing process of papers, not just pro- 
posals for funding — I mean, we are supposed to publish, many of 
us that work in science, our research, our results. It is very hard 
to get results published in the best journals if they are not of the 
highest quality. The same thing happens in the merit review proc- 
ess. You are not going to get funded unless you have the highest 
quality. So in a sense, we have like a voting system. Our colleagues 
actually vote about what we do, if they think it is good or not, as 
we get elected or reelected in Congress or not, I mean, the voters 
have to decide if we are doing a good job or not. As researchers, 
if we are doing a good job, our colleagues will vote that yes, get 
funded, yes, get published, yes, get evaluated for lectures and so 
on and so forth. So that is what I mean in the sense that we have 
checks and balances. 

When a recent result is wrong, the community will find out that 
it is wrong because when they try to reproduce that discovery and 
they cannot reproduce it, like cold fusion, for example, it just will 
die immediately. That is what I meant by checks and balances. 
That is how science advances, not in a straight line continuum. 

Chairman Brooks. Thank you for the additional information 
with respect to checks and balances. Do you have any specific sug- 
gestions on how we can strengthen or improve the process or sys- 
tem in order to help ensure that these scarce federal dollars are 
being spent on basic science or research rather than being affected 
by the criteria that are extraneous to those primary functions? 

Dr. Jose. Yes, of course. I think that, as I said, it is a human 
endeavor when people try to review proposals. As Representative 
Bucshon said, I mean, there is some element of human nature that 
interferes and how do we review proposals, who is reviewing the 
proposals, is the best investment that one can make or not. But we 
recognize that is a problem we have. We develop all kinds of checks 
and balances from the agencies, from you. You all the time are tell- 
ing us you have to convince us that you are doing the right thing, 
otherwise we won’t give you money to do the research, and it is not 
perfect and we keep changing it and correcting it as we go. 

Chairman Brooks. Well, my question is, do you have any specific 
suggestions on how to improve the process? 

Dr. Jose. Yes. I think that — let me — yes. I think that — actually 
I have thought a lot about that. There are blind types of reviews 
because often the name of the scientist plays a role in if it is going 
to get funded or not going to get funded. Perhaps even the institu- 
tion, perhaps the project and the area. I think that — let me see. 
How do improve the system? I think that we have to make sure 
that we choose the right reviewers. We have to make sure that 
there is no conflict of interest with the reviewers being part of the 
same group of people that are funding each other. Other than that, 
I think I have many things to say but it would take a while to de- 
scribe. 

Chairman Brooks. Well, thank you. Dr. Jose. You mentioned 
blind reviews. What are those? 

Dr. Jose. Well, for example, if you submit a proposal, you don’t 
know the name of the author of the proposal, the institution of the 
proposal, as you don’t know the name of the referees, and then you 
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will just blindly decide if this is really good or not whereas the way 
we do it now, I mean, we know the name of the reviewer. We can 
know if they are very good or not very good and decide — that may 
influence it a little bit. 

Chairman Brooks. Are you recommending that we have blind 
reviews? 

Dr. Jose. No, no. We have tried to do that. In fact, there is a 
journal trying to do that and they have mixed reviews, mixed re- 
sults. I mean, they tried to do that, not having the name of the au- 
thor, not having the name of the reviewers, and they just review 
it blindly. I am not sure that the end result was satisfactory as ex- 
pected. 

Chairman Brooks. Dr. Jackson or Dr. Yamamoto or Dr. Marrett, 
do you all have any specific suggestions that you think we should 
look at in order to try to improve the process? 

Dr. Jackson. Well, I do think — I will go back to what I men- 
tioned in my testimony about giving NSF program officers the abil- 
ity to triage proposals. At this point in time, all these proposals 
have to go through merit review. That taxes the scientific commu- 
nity, that taxes the program officers, and we need to make this — 
since we can’t fund all the proposals, and there are a lot of excel- 
lent proposals — there are more excellent proposals than we can 
fund — it makes sense for us to spend our time thinking about 
which of those excellent proposals we should fund rather than 
spreading our time across the whole gamut of proposals. 

Chairman Brooks. I am sorry. I understand what you have al- 
ready testified to, and my question should have been better. Do you 
have anything specific in addition to what you have already in- 
formed us of? 

Dr. Jose. I am sorry. I must say something about that, about 
what Dr. Jackson has. I mean, at the NSF, there are increasing 
number of proposals that have been submitted to the NSF for re- 
view and they have decided to assign to the university to have a 
limited submission-type thing. You can only submit two or three 
grants when there is several million dollars rather than having 10 
or 15. That is a change that was made and it is a change also that 
has been made in the NIH as well. 

Chairman Brooks. Thank you. 

Dr. Jackson, I don’t know if you have thought of anything in ad- 
dition to what you have previously shared with us? 

Dr. Jackson. It is a messy human process and it would be dif- 
ficult to make it more efficient. It is always something you are 
struggling to improve. 

Chairman Brooks. Well, I have to admit that when I heard it 
compared to the election process, that raised some concern. 

Dr. Yamamoto or Dr. Marrett, do you all have any other specific 
suggestions on what we can do to improve the process? 

Dr. Yamamoto. Let me make two comments that I want to frame 
as an expansion of what President Obama said when he addressed 
the National Academy of Sciences in April 2009. He said that a 
basic research project may not work for a year or a decade or ever, 
and it is for that reason that public funds, government funds, 
should support basic research because the private sector will al- 
ways underinvest in that side because when a project works, it 
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works for everyone. When it doesn’t work, of course, the private 
sector loses. Maintaining that basic research and engine is really 
critical, and so let me make one suggestion, which is an expansion 
of what I said about transformative research, and I will try to 
frame briefly why I think it really needs to be a distinct sector of 
research that is supported in this country, and then a second com- 
ment that really approaches this question of broader impacts. 

So transformative research, I suggested in my comments, and I 
will expand very briefly, should be viewed as a different kind of re- 
search. It is destructive of current views rather than extending 
them or deepening our understanding of current paradigms, and to 
do that requires that we recognize that special characteristic and 
have a review process that is very distinctive. I proposed the trans- 
formative research track at the NIH in 2006. It has been installed 
in a limited way. I chaired the first two rounds of review of those 
applications when they came in, and an essential feature of I think 
making that program successful is actually carving out a different 
mode of merit review, one that is driven by reviewers that are not 
deep content experts in the areas that are being proposed but rath- 
er are generalists who think deeply, who are able to recognize real- 
ly high-impact ideas when they see them and are able to celebrate 
the fact that those ideas may actually bring down their current 
ways of thinking about a process. 

Luckily, such scientists exist in this country. They can be found. 
They can be identified. They are willing to participate in this way. 
They are the kinds of people that are finally getting challenge 
grants from the Gates Foundation that are proposing — making pro- 
posals to the transformative track and are willing to review in that 
way. So it takes a different kind of review that recognizes that you 
don’t bring in content experts who made the paradigms because 
they are going to be immediately critical of someone who comes in 
and says I think the current paradigm is wrong. I could expand on 
that further. 

Let me move to the second point, something I didn’t mention, 
didn’t talk about at all, and in a sense, you could view this as being 
outside of the merit review process, but I think not, and that is the 
idea of recognizing, especially with this broader impacts mandate, 
that basic research has advanced to a point where it can increas- 
ingly be applied, and since I work in the biomedical sphere, I see 
this all of the time with the opportunities of our faculty in my own 
laboratory to be able to move fundamental research to application, 
but there is the so-called valley of death that I am sure you have 
heard about, that especially in the current economy has made it 
very difficult to be able to do the few experiments that are needed 
to make it evident to pharmaceutical companies and the biomedical 
sphere that there is a worthy investment there. I think that we 
have an opportunity to be able to extend the continuum to be able 
to build new interfaces between academic basic research funded by 
the NSF and other agencies and build public-private partnerships 
that can more effectively move ideas into application, and in that 
way be able to really directly approach this challenge of addressing 
broader impacts that the COMPETES Act really challenged all of 
us to be able to do. 

Chairman Brooks. Thank you. Dr. Yamamoto. 
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And Dr. Marrett, I don’t know if you have any suggestions. It oc- 
curred to me that if there some, you prohahly would have already 
implemented them as Deputy Director and Acting Director but 
nonetheless, if you have any other suggestions on how you think 
we can improve the process, please share it. 

Dr. Marrett. And I take your question to mean, what especially 
might the subcommittee do, and I have two recommendations 
there. One is to share broadly the understanding of the process of 
merit review. I am not sure how widely the process is understood 
by, I wouldn’t just say Congress, or general public, yet if we are 
drawing on public resources, we need to make sure that that proc- 
ess is understood. So this hearing represents a good approach for 
broadening knowledge about the process. 

The second recommendation I would make for the Subcommittee 
is, I hope you would be open to our sharing with you the experi- 
ments, the pilots that we are trying. We do say that there are 
areas where there can be improvement. Let me take the case of po- 
tentially transformative research. We are not all that satisfied but 
we are experimenting with, do we advance the potentially trans- 
formative research by identifying proposals once they have come in, 
by soliciting new ideas, by thinking of the sorts of experience from 
NIH. We are trying all of that. We would welcome then coming 
back as we have learned from those experiments to say more about 
what should be advanced. We are not quite ready yet to say these 
are the things that we think ought to be put in place. And finally 
there, I would say whatever, we think that at the core, that core 
has to be maintained. I mean by that core that commitment to 
merit and so whatever the sorts of changes, the changes are around 
the margins in many respects. They never should undermine the 
importance of relying on the meritoriousness of the idea and the 
approach that is being proposed. So thank you. 

Chairman Brooks. Thank you. Dr. Marrett. 

Mr. Lipinski, thank you for being patient. 

Mr. Lipinski. Thank you, Mr. Chairman. I think that — I am glad 
we went to a second round of questions because I think we really 
got into some really good ideas here about how, you know, to make 
improvements to the system, and I think actually I would ask — the 
Chairman asked Dr. Marrett if we could possibly do, maybe a tuto- 
rial, and get into how exactly this process does work, because we 
are talking about it here but even someone who has presented a 
proposal and has gotten a grant, I would like to really know better 
how the whole system actually works and maybe the opportunity 
for members to sit down and go through a tutorial. I think that 
would be a great idea for all of us. 

Chairman Brooks. That is a good suggestion, and I welcome it. 

Mr. Lipinski. Because I think there is a lot to the process, and 
we are talking about the process but I think there is a lot of details 
to the process and things that probably each of us doesn’t com- 
pletely understand and might be able to get a better understanding 
of that. 

Two things I want to get into. The first one. Dr. Jackson had — 
one of the main proposals you had to sort of lessen the work is for 
the program officers to be able to essentially triage and put aside 
proposals that come in that the program officer doesn’t see there 
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being merit to. I wanted to get a better understanding of what you 
are saying how that would be done, and then I will get Dr. 
Marrett’s response on that. Are you saying then that immediately 
set aside these proposals without doing any sort of response to 
them? Because right now all the proposals get a response from the 
reviewers. It would seem the only way you could save time really 
is if the program officer just put these aside without any comments 
and just send it back and say rejected right off the top. In that 
way, you would not be getting — you would not be giving the re- 
searchers any feedback at all. I was just wondering how exactly 
you saw that working. 

Dr. Jackson. Right, and you are right. That is the downside to 
triaging proposals that clearly do not — clearly aren’t up to snuff, so 
to speak, is that they miss the opportunity for a certain amount of 
feedback, and there are several ways you can handle that. Cer- 
tainly for the petroleum research fund at ACS, there are certain 
ways we deal with that. One is that you can ask for a review from 
the program officer who in our case and in NSF’s case is an expert 
in the field so that program officer does have to write up something 
about what is a problem with the project and this sort of thing. An- 
other thing that we do with the Petroleum Research Fund is to pro- 
vide workshops particularly focused at younger researchers, but 
anybody can come, that talk about what makes for a good proposal 
and so we are training the new generation and new people to the 
field how to write proposals, what we are looking for and what 
needs to be done, so we are doing this to counteract maybe some 
of the individual feedback that could be — that would be given if it 
was sent out to review. 

But you have to understand that a proposal goes out not just to 
one reviewer but to a number of reviewers, so there are possibili- 
ties too you could have; if you realized that this is not going to be 
funded, you could ask one reviewer perhaps to provide a review, to 
provide advice to the proposal writer, or there are a number of 
other things you can do that would decrease the amount of work 
for each proposal. But in the PRF, we recognize that these are just 
not going to get funded and so we spend a limited amount of time 
with them and try to do our education separately. 

Mr. Lipinski. Yeah, I understand the need to save time and I 
think we all want to find ways, because there is — reviewers are 
overburdened and it is hard to get people to serve in that capacity. 
I just want to give — and I am not, you know, being critical of that 
proposal. I want to see what Dr. Marrett would have to say about 
that. 

Dr. Jackson. Well, and we also — one more thing. We also en- 
courage those proposal writers to call our program managers and 
talk to them about their proposal so there can be some kind of ex- 
change and learning going on. 

Dr. Marrett. We are absolutely intrigued by anything that could 
help reduce the burdens on program officers, on reviewers. The 
number of proposals we are getting, the number continues to climb 
and it is very difficult. On the other hand, there is the kind of view, 
a very pervasive view that a part of the whole process is to help 
give good feedback, especially help in the cultivation of new inves- 
tigators and thus our program officers are hesitant simply to say 
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let us try to move this as quickly as possible. Because I think there 
is a responsibility to help people understand where the problems 
might have been, to help cultivate the ways in which they might 
develop stronger proposals. It is a dilemma then of how to have an 
effective, efficient strategy that is at the same time an educational 
strategy that we are all seeking to accomplish. 

So as I said, we welcome ideas but we are trying to examine 
them in the context of the larger kinds of things that we seek to 
do in the whole development of the science and engineering com- 
munity of the United States. 

Mr. Lipinski. And I wanted to go on to another, if the Chairman 
will indulge me. 

Chairman Brooks. Feel free. 

Mr. Lipinski. I wanted to ask about conducting the committee re- 
views in virtual environments. It is something I mentioned in my 
opening statement. I personally have had no experience at all. I 
have no avatar. I have not gotten into this so I can’t really speak 
from any expertise here, but I know this is something that has 
been looked at, and I was wondering if — I will start with Dr. 
Marrett. I want to ask all the other panelists about their opinions 
on using this as a way to make it easier in some ways in doing this 
in a virtual environment. You don’t have to fly people into Wash- 
ington to do this, and other possible merits to using this system. 
I would also think that the blind review process — and also having 
people come together in a way that they don’t know each other that 
could impact, change the dynamics of the group. So, Dr. Marrett, 
we will start with you. 

Dr. Marrett. Yes, we are looking very much at virtual environ- 
ments. There are experiments that have been taking place, and 
these environments, thinking of virtual panels especially, we have 
been thinking of them for two reasons of reducing some of the cost, 
the cost for reviewers, and for potentially expanding the pool of re- 
viewers. In other words, there are some people who would find it 
difficult to do the traveling since most of these are here in this 
area. We thought this would be a way then to bring more people 
into the process. 

What we are looking at, what are the sorts of things that have 
to be taken into account. There are matters of security, for exam- 
ple, that you have to make sure that you have been abiding by. 
You raised the questions about the kind of interpersonal dynamics. 
There, we are actually drawing some of the research being spon- 
sored by our Directorate for Social Behavioral Economic Sciences 
where there are studies of human dynamics, enough for us to un- 
derstand how those dynamics might come into play and actual 
studies specifically of virtual environments. All of this then we 
think we need to take into account and that is why we are experi- 
menting but we are trying to bring the knowledge to bear as much 
as we can to ask what would be the consequences and where there 
might be some savings but where there might be some losses if we 
are not careful about how we try to institute the idea of virtual en- 
vironments. 

Mr. Lipinski. Any other panelists have any opinions here? I will 
start with Dr. Jose. 
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Dr. Jose. Yeah, let me say that since I am quite old and I was 
funded for many years, I can see the evolution of how proposals are 
reviewed by the NSF. For a long time, there was only by mail re- 
view, and they only waited until the reviews were done and they 
analyzed what the kind of grade each one of the reviewers gave. 
Then they went into having a combination of having mail and 
panel reviews. And I can tell you that by having done both, there 
is a significant change in the actual assessment of the proposals 
that arises from meeting with people live than just from reviewing 
it by yourself 

Okay. Now, with the virtual analysis they have to do, it is very 
important to know what kind of human dynamics are going to in- 
fluence to some extent the reviewing process, and the reason I am 
saying this is that one of the committees that I was a member of, 
the mail reviewers gave much better grades to the proposals than 
when we sat down and discussed the proposals and interacted and 
listened to everybody else live, okay, so what effect would it have 
to have it virtually and not face to face, one has to wonder. But 
it is course a very useful and important thing to explore, yes. 

Mr. Lipinski. Who else? Dr. Jackson? 

Dr. Jackson. Living in Albuquerque, New Mexico, which seems 
to have no direct flights to anywhere, the idea of having virtual 
meetings is very exciting to me and to my colleagues because we 
are more likely to be able to be involved in these sort of — you 
know, become more — participate more in these sort of activities. 
But also because of that, I have done a lot more of this sort of vir- 
tual meetings and this sort of thing and I would say that yes, per- 
haps when you first start doing it, it is a little unusual but it is 
remarkable how with time you can become very accustomed to it, 
and I highly encourage NSF to consider these sorts of possibilities 
because I think the kind of expertise they could tap into will only 
become greater and richer. 

Mr. Lipinski. Yeah, obviously I brought up avatars. You don’t 
have to do it that way, but Dr. Yamamoto? 

Dr. Yamamoto. I agree. I think there are two drivers for this in- 
creased — rapid evolution toward electronic environment for review. 
The first is that those of us who didn’t grow up using those media 
are disappearing and being replaced fortunately by people who 
have screens in front of them their entire lives and are very com- 
fortable working in that environment. And the second is the tech- 
nology development is marching along very quickly and there are 
electronic video conferencing modes that you are looking across the 
room at a screen and it looks exactly like we are looking across the 
room at you. And so the opportunity for interactions, which I agree 
with Dr. Jose, is essential in being able to come to this sort of 
agreement about the scientific merit of an application that can 
really occur increasingly seamlessly in this sort of environment. 

Let me make one further comment, because you asked. Congress- 
man Lipinski, about this idea of blind reviews that Dr. Jose had 
mentioned, and just say there again here is the spectrum of experi- 
ments ongoing in different agencies. You may know that the How- 
ard Hughes Medical Institute does actually the opposite of a blind 
review. In fact, rather than choosing scientific projects as the agen- 
cies that we all represent and are working and do, they simply 
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choose people, and so the identification of the person is the whole 
hallgame for the Howard Hughes Institute. The Gates Foundation 
in their initial round of reviews for their Grand Challenges grants, 
which I participated in several times, is blinded to the identity of 
the applicants, and after they make it through the first round, then 
that identity is revealed, and then NSF and NIH, the federal agen- 
cies that I am familiar with, the identity is always known. 

There is a risk of conflict of interest or other kinds of biases that 
can be introduced at that point, but let me say that having been 
experienced in review for at least NSF and NIH extensively that 
I don’t actually see that actually occurring, and instead knowing 
the identity of the investigator, in fact, building it in as an explicit 
criterion for the merit review process, turns out to be very impor- 
tant. There have been studies done and arguments made that in 
fact the strongest predictor of success of a given scientific proposal 
is in fact the track record of the investigator who is making the 
proposal. So in that sense, knowing that person, their past capacity 
for doing innovative work — and let me just insert here that this 
does not mean — ^because very often that statement that I just made 
raises concern that this would bias against young investigators. In 
fact, it does not, and I think the clearest view of that is that I have 
been involved, and I am sure my colleagues have as well, for years 
in making judgments about graduate fellowships, postdoctoral fel- 
lowships, young trainees who have very little in the way of conven- 
tional track records but it is very easy, I think leaving those meet- 
ings, those of us who are on those panels, feel very confident that 
we have been able to make the right choices for identifying the best 
candidates based on what they have done before, even thoug:h they 
have been in a training mode and very different from being an 
independent investigator, and data prove that out that those people 
that win such fellowships and awards go on to be able to do very 
good work. 

And so my view is that having that identify early on at least — 
I think the Gates Foundation strategy actually works quite well — 
but being able to unmask those individuals early on, to be able to 
see their track record and build it into the merit review process is 
important. 

Mr. Lipinski. Well, let me throw my own two cents in here. I am 
not always certain from — I think it depends on the field possibly. 
In my background as a political scientist, I was not always con- 
vinced that it was helpful to the advancement of knowledge to 
know who was proposing something because sometimes I think 
that had an influence that wasn’t good on review processes, and I 
am talking more about journal reviews. But that is my own two 
cents. It is very interesting to hear, though. Dr. Yamamoto’s and 
everyone else’s experiences with this, and I thank all of you very 
much and I especially thank the Chairman for his indulgence here 
and for this hearing. 

Chairman Brooks. My pleasure, but we are not finished yet. I 
am going to follow up on something that Mr. Lipinski noted, and 
that is, we have talked about the virtual panels but what is the 
cost of in-person panels? 

Dr. Marrett, do we have any kind of transportation costs or hotel 
costs or is there a budget line item, or how much does it cost per 
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panel or how many panels are there per year? Can you share some 
information with us about that? 

Dr. Marrett. We will get that information to you. We actually 
have done the calculations of what the costs are associated with 
panels, and those do include the travel costs, of course, and there 
are hotel issues there, and we had started as we were looking at 
what would he the savings from virtual panels, and I should men- 
tion that in the case of avatars, we have had reviews done in Sec- 
ond Life. But in terms of the sorts of costs associated with virtual 
panels, we are not quite sure because there are other kinds of 
things. We just haven’t explored enough now. We can’t do the com- 
parative information but we can get to you how many panels, what 
the cost is of an average panel for the National Science Foundation. 

Chairman Brooks. And if you can do a comparison format, too, 
I would anticipate that the virtual panels are significantly less ex- 
pensive. But if we have some kind of comparison model, that would 
be beneficial. 

Dr. Marrett. We will try that, but why I say it is not necessarily 
going to be much cheaper because there is the question of the tech- 
nology that we might have it on our side, but it is also got to be 
at the other side, and one of the things that has been talked about 
will be using regional panels where there will be cost of travel to 
the regional panels as well, the enhanced security costs that come 
into play, so we are not quite sure at the outset that there would 
be lots of savings that could happen over time in a number of ways. 
That is why I said we won’t have for you — it will be very rough 
kinds of estimates when it comes to virtual panels but we can tell 
you about the investments currently made in the in-person panels. 

Chairman Brooks. Thank you. Dr. Marrett. 

As a short concluding statement, I don’t know if you all have ac- 
cess to this, perhaps Dr. Marrett already has it, but some kind of 
compilation of success stories with NSF, the kind of information 
that this Subcommittee can share with the public or the Members 
can be familiar with so when asked, we can share it with the pub- 
lic. I am sure you are all familiar with the relatively famous 
shrimp on a treadmill commercial that has been running over the 
last month or two, and I am afraid that that may give the general 
public some dissatisfaction with the way their tax dollars are being 
spent, so I think it is important to also have information that we 
can share with the public that shows success stories and the kinds 
of returns on the investments that we are asking taxpayers to 
make. 

Having said that, I want to thank the witnesses for their valu- 
able testimony and the Members for their questions. If the Mem- 
bers of the Subcommittee have additional questions for the wit- 
nesses, you are free to submit those, and we will ask you to re- 
spond to those in writing. The record will remain open for two 
weeks for additional comments from the Members. 

The witnesses are excused and this hearing is adjourned. 

[Whereupon, at 11:43 a.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by Dr. Cora Marrett, 

Deputy Director, National Science Foundation 

Questions submitted by Chairman Mo Brooks 

Ql. As requested during the hearing, please provide the total amount of federal 
funding awarded to those proposals rated from “Poor” to “Good or Very Good” for 
FY 10. While you touched on it briefly at the hearing, please also expand on why 
those proposals received funding over proposals rated “Very Good to Excellent” and 
“Excellent. ” 

Al. All funded proposals are determined to be highly meritorious based on a com- 
bination of reviews by individuals, panel deliberations and program officer evalua- 
tion. On average, NSF proposals are reviewed by four to six individuals, depending 
on the type of review. All reviewers are chosen for their specific expertise related 
to the subject, and the collection of persons brings different points of view to the 
decision-making process. When the average reviewer score is in the “good” range, 
it often represents a split of “excellent” or “very good” reviews with some “fair” or 
“poor” review scores that lowered the average. It is important to note that the pro- 
posal rating data included in the annual NSB Merit Review Report reflects proposal 
ratings before panel deliberations and, therefore, not the final panel evaluation. The 
panel evaluation is based on a thorough discussion of the proposal’s strengths and 
weaknesses in the context of the full set of proposals being reviewed; this discussion 
forms the basis for placing a proposal in a particular category. These in-depth dis- 
cussions can often clarify perceived weaknesses and result in a proposal being rec- 
ommended for funding despite the initial average review score. Likewise, some pro- 
posals with high average review scores are not recommended by panels as a result 
of a detailed discussion that uncovers weaknesses that might not have been re- 
flected in the initial reviews. 

The expertise of the NSF Program Officer making the final recommendation is 
also an important voice in the process. Program Officers take into consideration 
other factors that might not have been considered by expert reviewers. For example, 
proposals for innovative new ideas often use unproven methods or techniques that 
might be considered risky by reviewers and panelists. Risky proposals often result 
in transformative research that accelerates the pace of discovery. Although Program 
Officers consider concerns about risk expressed by panels, they also see the value 
of funding potentially transformative research. Proposals that do not review well at 
panel because the methods are unproven or risky can be given small awards to 
allow enough work for a “proof of concept.” Program Officers will also consider 
broader impacts that might not be obvious to reviewers, such as an infrastructure 
need that will serve a large number of people. There are also many dimensions of 
portfolio balance that influence the final recommendation. In addition to maintain- 
ing a diverse scientific portfolio. Program Officers strive to fund proposals from di- 
verse institution types across the U.S., and from both young and experienced inves- 
tigators. 

As explained above, the reviewer rating data reported in the Merit Review Report 
are only initial reviewer ratings, which is just the beginning of the merit review 
process leading to a final dertermination of whether any given proposal should be 
funded. Initial reviewer ratings do not reflect panel deliberations or Program Officer 
input. In FY 2010, NSF funded approximately $46K in proposals initially rated as 
poor, $21M in proposals initially rated as fair, $818M in proposals initially rated 
as good, and $1.6B in proposals initially rated as very good. Following panel discus- 
sion and analysis, all of these proposals were determined to be highly meritorious 
noth withstanding their initial rankings. 

Q2. In your testimony, you described experimenting with innovative approaches to 
identify potentially transformative research. Please expand on the “ideas factory 
sandpit” approach and tell us what you are learning from it and other novel ap- 
proaches. 

A2. NSF has experimented with an approach to identif 3 dng potentially trans- 
formative high-risk research that it is now calling “Ideas Lab.” The Ideas Lab is 
closely modeled on the “sandpit” process developed by the UK’s Engineering and 
Physical Sciences Research Council (EPSRC). The essential element of the Ideas 
Lab is an intensive interactive residential workshop involving 20-30 participants, 
with the aim of developing bold, often risky, new approaches to grand challenge 
questions in areas that could benefit from creative “out-of-the-box” thinking. A fun- 
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damental aspect of the EPSRC sandpit that has been incorporated into the Ideas 
Lab is the use of a highly multidisciplinary mix of participants (including active re- 
searchers from diverse fields and potential users of research outcomes) to address 
specific research challenges. A description of the process used follows. Slight vari- 
ations should be anticipated as NSF gains experience with the process and as it is 
adapted to different topics. 

To identify potential participants, a solicitation is issued that includes an open 
call for participants. Interested individuals submit short preliminary proposals that 
include concise descriptions of their pertinent experience and expertise as well as 
their communication skills, collaborative activities, and creative abilities. A panel of 
reviewers evaluates the applications and identifies a pool of potential participants 
from a range of disciplines and backgrounds who have high potential to work at the 
interface between disciplines and to develop new and highly original research ideas. 
NSF Program Officers make the final selection from the pool to ensure a diverse 
mix of participants. Industrial psychologists provide advice that guides but does not 
decide participant selection. 

During the multi-day Ideas Lab workshop, participants interact in unconventional 
new ways to develop innovative research project ideas on the selected topic area. 
Professional facilitators, experienced in sandpit-like activities, integrate creative 
problem-solving techniques, iterative project-development activities, and real-time 
peer review by both participants and a resident panel of experts (called the mentor 
group) to advance the most innovative ideas. Outcomes at the end of the workshop 
are research project concepts that vary in scale and scope in addressing the grand 
challenge topic of the Ideas Lab. At the end of the Ideas Lab, the panel of reviewers 
provides a consensus report summarizing its evaluation of each project concept. 
Based on this review, the Program Officers invite the submission of full proposals 
for some, none, or all of the project concepts. The invited groups have six to eight 
weeks to submit full proposals, which are then reviewed by the same panel of men- 
tors using NSF’s two merit review criteria. Based on that review, NSF then makes 
a decision whether to fund some or all of the proposals. Taking part in the Ideas 
Lab does not mean that the participant is guaranteed to be funded under an award 
resulting from the Ideas Lab process. 

Experimentation with the Ideas Lab is still at an early stage. A total of four Ideas 
Labs have been conducted to date. The first three resulted in 12 awards and the 
fourth is currently awaiting full proposal submissions. Feedback from participants 
in the Ideas Labs has been positive. However, the resulting funded projects are still 
in their beginning phases. As they progress, NSF will look at the outcomes of these 
projects to evaluate whether they resulted in transformative research. 

Q3. Researchers will send in proposals whenever they have an idea that they would 
like to have funded. However, NSF also puts out solicitations for specific areas of 
research. Please explain how decisions are made on what types of research areas war- 
rant a specific solicitation from the Foundation? What happens if the Foundation 
does not receive high quality proposals for a solicitation? Do you pick from what you 
have or do you rework the solicitation? 

A3. Solicitations are formal NSF publications that encourage the submission of 
proposals in specific program areas of interest to NSF. Solicitations are generally 
more focused than program announcements, and normally apply for a limited period 
of time. Ideas for new solicitations can be initiated within Divisions by Division Di- 
rectors, at the Directorate level by Assistant Directors, or by groups of Assistant Di- 
rectors who see the need for a new cross-cutting activity. Program Directors also 
commonly suggest ideas for new initiatives. The initial spark for a new initiative 
often comes from interaction with the scientific community through scientific meet- 
ings or other communications. When an idea for a new solicitation is suggested, a 
working group is formed that includes Program Officers who are expert in the re- 
search area. The working group collaborates on a detailed plan for the new solicita- 
tion, which is then discussed and reviewed by various levels of leadership before ap- 
proval. 

Specific factors that are considered when deciding whether to develop a new solici- 
tation include the following: 

• the intellectual reason for the Program, activity, or initiative; 

• whether the new activity(ies) will generate sufficient interest in the targeted 
community; 

• whether the Program, activity, or initiative is new, how it supports the long- 
range goals of the Directorate and/or NSF; 

• whether the size of the effort justifies a separate announcement and/or competi- 
tion; 
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• the total funding available for the proposal competition, including estimated 
proposal receipts and anticipated number of awards and funding levels; 

• cross-Directorate participation (and implications) in the Program. 

Program Solicitations often specify submission limits, award conditions or report- 
ing requirements, and provide supplemental proposal preparation guidance in addi- 
tion to what is in the Grant Proposal Guide. Program solicitations also provide spe- 
cific review criteria when reviewing proposals. In cases where the Foundation does 
not receive high quality proposals for a solicitation, we decline the proposals that 
are not of high quality. The solicitation could be revised and recompeted to attract 
high quality proposals. Because NSF funding opportunities generally attract more 
high quality proposals that we can fund, this would be a rare occurrence. 

Q4. After reviewing the flow chart for the proposal and award process and timeline, 
the Directorate Assistant Director seems conspicuously absent from the process. 
Please describe what the role and responsibilities of the Assistant Director are in the 
funding process, both from a programmatic and overall agency funding priorities 
perspective. 

A4. While Assistant Directors (ADs) are not involved in the day-to-day review and 
processing of proposals submitted to the Foundation, as described in the NSF Grant 
Proposal Guide Exhibit III 091 (http: ! Iwww.nsf.gov ! pubs ! policydocs ! pappguide ! 
nsfl 100 1 gpg -3exl.pdf), they fulfill a vital role in the overall funding process. (See 
the attached referenced flowchart.) 

ADs are knowledgeable about the award portfolios in their directorates, but they 
are not involved in the decision-making process itself, because their role is to set 
the vision and strategic goals and objectives for the divisions/offices that report to 
them. 

Assistant Directors also play an important role in the formal reconsideration proc- 
ess. If a PI is dissatisfied with the explanation they receive for why a proposal has 
been declined, he/she may request a reconsideration of the decision. ADs/Office 
Heads are responsible for responding to these requests, and review the proposal 
record to determine whether NSF’s review of a declined proposal was fair and rea- 
sonable, substantively and procedurally. If they were involved in the decision-mak- 
ing process, the would have a conflict of interest in responding to any official recon- 
sideration request. 

Q5. How does the Foundation leadership ensure that Program Officers “produce and 
manage a balanced portfolio of awards that address a variety of considerations and 
objectives” as the FY 10 NSB Report states'? 

A5. Portfolio balance is reviewed at a variety of levels at different times during 
the decision-making process. Program Officers consider many dimensions of portfolio 
balance when they are making decisions about what proposals should be rec- 
ommended for awards. Some of the factors that are considered include: balance 
across disciplines and subdisciplines, award size and duration, awards to new inves- 
tigators, geographical distribution of awards, awards to different types of institu- 
tions, innovative/potentially transformative projects, projects with elements of risk, 
inter- and multidisciplinary projects, projects that integrate research and education, 
and projects that are relevant to agency mission or national priorities. Division Di- 
rectors review the recommendations by Program Officers for portfolio balance before 
they concur with the award recommendations. Portfolio balance is also reviewed by 
our Committees of Visitors who review programs at three-year intervals. Some pro- 
grams also contract for external evaluations of their portfolio periodically to inform 
how they might make changes to their programs. The results of both COV reports 
and external portfolio analyses are reveiwed by Directorate Advisory Comittees. 

Q6. According to the FY 10 Board Report, NSF awarded approximately five percent 
of its annual budget to federal agencies and laboratories. What kinds of awards were 
these and did they go through the formal merit review process? 

A6. The 2010 Merit Review Report to the National Science Board reported that 
NSF funded $351. 2M in awards to Federally Funded Research and Development 
Centers (FFRDCs). The majority of this funding went to two organizations that 
build and manage large astronomy facilities for university consortia: Associated Uni- 
versities Inc. (AUI) and the Association of Universities for Research in Astronomy 
(AURA). AUI received $111M in funding associated with the National Radio Astron- 
omy Observatory, the Atacama Large Milimeter Array (ALMA), and other related 
projects. The Association for Universities for Research in Astronomy (AURA) re- 
ceived $234.3M for a number of projects including building the Advanced Tech- 
nology Solar Telescope and the Advanced Technology Solar Telescope and operations 
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and management of the Gemini Observatory, the National Optical Astronomy Ob- 
servatory, and the National Solar Observatory. In addition, $0.4M in funding went 
to fund several much smaller projects through another FFRDC, Aerospace Corpora- 
tion. Proposals submitted to NSF by FFRDCs go through the same merit review 
process as other proposals. The large awards for building and operating large facili- 
ties go through a very lengthy and detailed review process that includes site visits, 
cost reviews, design reviews, and approval by the National Science Board. 

The $351.2M reported in the Merit Review Report also includes $5M in contracts 
to fund the Science and Technology Policy Institute (STP) operated by the Institute 
for Defense Analyses. STPI provides rigorous and objective analysis of science and 
technology (S&T) policy issues for the White House Office of Science and Teehnology 
Policy (OSTP) and other offices and councils within the Executive Branch of the 
U.S. Government and federal agencies. IDA was selected to operate STPI in 2003 
following a competition and undergoes reviews at five-year intervals. 

Note that the funding to FFRDCs described in the FY 2010 Merit Review Report 
did not include contract funds to or from other federal agencies through interagency 
agreements. 

Q7. What kind of peer reviewers are coming from industry, non-profits, and govern- 
ment Do they all have Ph.D.s? What role does a panelist from the government play? 
What qualifieations do they have? 

A7. Reviewers are chosen for their expertise in areas covered by the proposals that 
they are asked to review. For research proposals, reviewers are typically researchers 
in domains of science relevant to the topic of the proposal. In the review of proposals 
for facilities, in addition to the reviewers who can provide input on the research im- 
pacts, technical feasibility and soundness of the faciltity’s design. Program Officers 
may also include reviewers with expertise in other relevant fields such as project 
management, systems engineering, complex acquisition processes, architectural de- 
sign, etc. 

In many scientific and engineering disciplines, some of the leading researchers 
work in industry, non-profits, government laboratories and FFRDCs. Examples in- 
clude computer science and a number of others. Such researchers are very much the 
peers of their academic counterparts and are included in NSF’s pool of peer review- 
ers. Some of these researchers may have spent part of their research careers in aca- 
demia and some in industry or an EFRDC, allowing them to bring important per- 
spectives on the state of the art in the different environments and their potential 
broader impacts. These reviewers tend to have the qualifications that are typical for 
the research communities to which they belong. In many disciplines, this is often 
a Ph.D., although occasionally it is simply long experience doing cutting-edge re- 
search. Typically, what signals the expertise of a researcher is his or her record of 
research achievement, including significant publication in peer-reviewed journals 
and conference proceedings. 

Reviewers from government and industry are often more familiar with project 
management and complex acquisition processes than some of their academic coun- 
terparts, and so such individuals are sometimes asked to bring this expertise to re- 
view teams looking at proposals for research facilities. Such individuals may or may 
not have Ph.D.s. 

In general, what NSF looks for in its choices of reviewers is expertise in the topics 
under review. 

Q8. How does the Foundation train reviewers to prevent the phenomenon of implicit 
bias? 

A8. The frontline of the merit review process are the approximately 520 NSF Pro- 
gram Officers (POs) who select experts who can provide the information needed to 
make a recommendation in accordance with the National Science Board (NSB) ap- 
proved criteria for selection of projects. Program Officers are trained on conflicts of 
interest, the importance of getting a diversity of perspectives, and guarding against 
the influence of subjective or biased input. 

Proposals submitted to NSF receive rigorous and objective treatment and POs en- 
sure that this takes place. Proposals are evaluated by independent reviewers con- 
sisting of scientists, engineers, and educators who do not work at NSF or for the 
institution that employs the proposing researchers. NSF selects the reviewers from 
among a pool of experts in each field, and their evaluations are anonymous. On av- 
erage, about 50,000 experts give their time to serve on review panels each year. POs 
ensure that there is diverse representation within the review group. The goal is to 
achieve a balance among various characteristics, including type of organization rep- 
resented, reviewer diversity, age distribution and geographic balance. 
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The reviewer’s job is to provide advice to NSF on which projects are the highest 
priorities. This competitive process ensures that many voices are heard and that 
only the best projects make it to the funding stage. When someone is asked to re- 
view a proposal (either as an ad hoc or panel reviewer), they are provided with in- 
formation on the confidentiality of the process and the potential for conflicts of in- 
terest. Panelists sign a “Conflict-of-interests and Confidentiality Statement” when- 
ever they participate in a panel. For ad hoc reviewers, by submitting their review, 
they are acknowledging that they’ve been informed of such policies. Again, NSF POs 
are responsible for assuring that appropriate, qualified merit reviewers are selected 
and the entire process is overseen by Section Heads and/or Division Directors who 
supervise the Program Officers. 

Q9. The 2010 reauthorization of the America COMPETES Act required the Founda- 
tion to “Apply a Broader Impacts Review Criterion to achieve” various goals. Wit- 
nesses at the hearing raised some concerns with the draft criteria that is currently 
being weighed by the Board. Have the goals, now specified in statute, been considered 
in the past when making funding decisions? Are the peer reviewers taking these goals 
into consideration during their review or are the Program Officers simply tasked 
with this responsibility? Based on the work being conducted by NSB and NSF and 
your experience with the merit review process, is the legislative requirement 
achiveable and is it necessary? 

A9. NSF strives to invest in a robust and diverse portfolio of projects that creates 
new knowledge and enables braekthroughs in understanding across all areas of 
science and engineering research and education. To identify which projects to sup- 
port, NSF relies on a a merit review process that incorporates consideration of both 
the technical merits of a proposed project and its potential to contribute more broad- 
ly to advancing NSF’s mission “to promote the progress of science; to advance the 
national health, prosperity, and welfare; to secure the national defense; and for 
other purposes.” In 1997, these considerations were put into action through the two 
primary merit review criteria of Intellectual Merit and Broader Impacts. Each re- 
viewer must consider, and address, both merit review criteria for eacb proposal. 

As noted in your question, the importance of incorporating consideration of poten- 
tial broader impacts in deciding which projects to fund was re-emphasized in the 
America COMPETES Reauthorization Act of 2010. Having the reinforcement of 
Congressional support on the fundamental nature of the Foundation’s Organic Act 
is always an important, and appreciated, development. However, there is a danger 
of viewing the Broader Impacts criterion as a “one size fits all” checklist, which 
would be a mistake. 

This COMPETES Reauthorization identified a number of societally relevant out- 
comes that may result as a consequence of NSF-funded research. Stated more 
broadly, these outcomes include (but are not limited to) increased participation of 
women, persons with disabilities, and underrepresented minorities in STEM; im- 
proved STEM education at all levels; increased public scientific literacy and public 
engagement with science and technology; improved well being of individuals in soci- 
ety; development of a globally competitive STEM workforce; increased partnerships 
between academia, industry, and others; increased national security; increased eco- 
nomic competitiveness of the United States; and enhanced infrastructure for re- 
search and education. These represent examples of societally relevant outcomes. The 
NSF will strive to clarify that these examples should not be considered either com- 
prehensive or prescriptive, and that investigators may include appropriate outcomes 
not covered by these examples. 
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Responses by Dr. Keith Yamamoto, 

Vice Chancellor for Research, University of California San Francisco 

Questions submitted by Chairman Mo Brooks 

Thank you for these insightful questions, Chairman Brooks. 

Ql. In basic science research, when one cannot possibly know what the outcome may 
be, how can one establish what the broader impacts will be? 

Al. Basic science research, hy definition, is untargeted, driven only hy the curi- 
osity, intuition, background and expertise of the investigator(s) who define a ques- 
tion and develop a plan to answer it. For such investigations, it is impossible to as- 
cribe broader impacts at the time of merit review. Only after the work has been 
completed, often long after its completion, can the wisdom of retrospection be ap- 
plied to define broader impacts. Thus, inclusion of the broader impacts criterion in 
the merit review of grant applications is misguided and damaging to the integrity 
of the merit review process. 

Q2. Having sat on review panels, please tell me how heavily the broader impacts of 
a proposal are weighed when rating the proposals you were considering? Did the 
evaluation of broader impacts have any effect on the evaluation of the intellectual 
merit of the proposals? 

A2. My experience on NSF review panels predated the 1998 development of the 
broader impacts criterion, so I cannot provide experience-based perspective here. 
However, as indicated in my testimony, any departure from singular focus on sci- 
entific merit, as mandated by this assessment of broader impacts, erodes the evalua- 
tion of the intellectual merit of the proposals. 

In this regard, I wish to comment in particular on the point raised by Ms. Jackson 
that the broader impact criterion provides a metric to distinguish proposals that are 
equally meritorious on scientific grounds. She viewed this metric as “an important 
role for the broader impacts criteria and should be kept as such because there are 
clearly far more proposals that are excellent than can be funded.” With due respect, 
my opinion is sharply different. Application of this approach would discriminate 
against true basic science research as defined in my response to your first question 
above, and favor funding of work with a clear application. To the extent that this 
evaluation metric would become known in the scientific community, this approach 
would lead to a decline in basic science proposals. Fundamental discovery remains 
absolutely essential for scientific progress, and would suffer significantly from nega- 
tive discrimination applied at the merit review stage. 

In NIH peer review, as noted by Dr. Marrett, this conflict is avoided by placing 
broader impact considerations in a separate review process carried out by separate 
review committees, thus maintaining the singular focus of merit review. 
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Responses by Dr. Nancy B. Jackson, 

President, American Chemical Society 

Questions submitted by Chairman Mo Brooks 

Ql. In basic science research, when one cannot possibly know what the outcome may 
be, how can one establish what the broader impacts will be? 

Al. At the National Science Foundation (NSF), all proposals are evaluated for in- 
tellectual merit and broader impacts. Two major determinate factors in estimating 
the long-term broad impacts of scientific proposals are the amount of resources 
available to fund grants and the standards of the Request for Proposal (RFP). 

NSF receives many more meritorious proposals than it could ever fund. While a 
proposal with weak intellectual merit has no hope of getting NSF funding, many 
proposals are rated “excellent” with strong intellectual merit and still do not get 
funded because of the stiff competition. The broader impacts criteria allow decision 
makers to gauge which research is the most urgent or has the greatest relevance 
to improving the quality of life. This merit review process enables NSF to ensure 
that precious R&D money goes only to the most relevant R&D needs. 

As anyone with a retirement fund has been told, managing a portfolio is critical 
to its long-term strength. Financial advisors stress that it’s important to find the 
right balance between solid performing stocks and riskier investments that may pro- 
vide higher returns. Managing a research portfolio is similar. 

The research manager, whether working in industry or at a government agency, 
strives to find the right balance between science that will deliver steady advances 
and ideas that are out of the box, but could result in game-changing developments. 
This point is especially important in times of restricted funding. It’s human nature 
to make more conservative choices and be risk-aversive when times are tough. Ex- 
panding resources allows grant reviewers to “take a chance” on promising but less 
established scientific ideas that may result in a major scientific leap when compared 
to a safer but more conservative proposal. 

For every 100 grant proposals a program officer reviews, a small number (perhaps 
10) will be of such high quality that it is obvious they should be funded. Another 
50 proposals will be recognized as clearly not competitive; however, they still must 
be addressed through the process. An agonizing choice must then be made over the 
40 proposals in the middle. These include proposals that may be considered excel- 
lent or very good. In fiscal year 2010, the NSF award rate was 23 percent. In our 
analogy of 100 proposals, this would mean that 13 out of the remaining 40 would 
be funded. 

Unfortunately, it has proven difficult to pinpoint from where the next game- 
changing scientific moment will occur. In 1994, when NSF funded the Stanford Inte- 
grated Digital Library Project, the Web was a tiny portion of the Internet, which 
was dominated by file sharing and gopher sites. This innovative grant to fund re- 
search into developing methods for searching Internet databases eventually lead to 
the creation of a $200 billion dollar company (Google, Inc.), which revolutionized al- 
most every aspect of the way humans interact with each other and information. 
However, it would have been difficult to pinpoint that particular project as the most 
promising grant NSF would fund in 1994. 

Similarly, Dr. Robert Grubbs won the 2005 Nobel Prize in chemistry for his work 
in olefin metathesis. This complex organic reaction essentially allows molecules to 
swap components and is broadly used today in the pharmaceutical industry. The 
basic understanding of olefin metathesis dates back to the 1950s; however, at that 
time it was deemed expensive and dangerous for industrial use. For four decades 
corporate and academic researchers labored to refine and improve the reaction, 
which did not become industrially productive until the late 1990s. NSF was a major 
contributor to Dr. Grubbs’ work, which is revolutionizing the way drug companies 
improve the environmental footprint of chemical reactions. Without NSF’s long-term 
dedication to the rigor and quality of Dr. Grubbs’ (along with many others) work, 
safe and economically viable olefin metathesis may not have occurred. 

These two examples (one based on a single grant that revolutionized the informa- 
tion age, the other, the result of four decades of research by multiple teams of re- 
searchers) show the need for sustained, predictable research funding. Ensuring a 
steady stream of research funds would allow NSF to fund not only the most rigorous 
and reliable research, but also take risks in cutting-edge ideas based on promising 
data. By doing so, the government would be encouraging the broadest possible im- 
pacts of taxpayers’ precious dollars. 

One of the reasons why the merit review process is so successful is because it 
draws from the collective wisdom of the scientific community. Many NSF personnel 
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come directly from the scientific community and return to their research institutions 
at the end of their two- or three-year rotations. Rel3dng on rotating directors means 
the managers are up to date on the most recent scientific developments. The panels 
that perform the peer review of proposals are fellow researchers in the field, and 
as such, are also up to speed on the latest developments. This scientific community 
service, whether performed by grant proposal reviewers or NSF program officers, is 
an integral part of scientific culture. Many scientists dedicate their time in this way 
because it provides an opportunity to remain in touch with and influence the cutting 
edge, as well as because they understand that the system only works if everyone 
volunteers to play their part. In a way, it is the science community’s way of “pa3dng 
it forward.” 

The broader impacts criteria include considerations about whether the research 
proposal would broaden underrepresented minorities’ participation in science, 
strengthen U.S. infrastructure, improve national security, or foster innovation. In- 
cluding some of these impacts is the result of language in the America COMPETES 
bill enacted last year. The broader impacts criteria take into consideration which 
research is the most urgent or has the greatest relevance to improving the quality 
of life. The broader impacts criteria enables NSF to choose between meritorious and 
even more meritorious proposals and is a way to ensure that precious R&D money 
goes to the most pressing R&D needs. 
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Responses by Dr. Jorge Jose, 

Vice President for Research, Indiana University 

Questions submitted by Chairman Mo Brooks 

Ql. In basic science research, when one cannot possibly know what the outcome may 
be, how can one establish what the broader impacts will be? 

Al. Thank you, Chairman Brooks, for the question. As you correctly note, in basic 
science research, it is not possible to know in advance what the outcome of research 
will be. We pose a question for which we would like to find an answer, and we lay 
out a research plan that in principle should lead us to an answer. The research plan 
is based on prior knowledge and an educated hypothesis of the expected results. A 
researcher may not know for certain that a particular plan will yield the results 
they hope for, but they do know aspects of a plan that is likely to do so. Reviewers, 
in fact, will base their review on whether the hypothesis and the research plan seem 
to be an appropriate way to ascertain the ansers to the question being posed. In 
the course of an investigation, other questions or an unexpected discovery may arise 
that may lead the researcher to significantly change the direction of the research. 

My sense is that a similar awareness of cause and effect can shape our expecta- 
tions for what the broader impacts of any given research project might be. Take, 
for example, the broader impact goal of expanding the participation of underrep- 
resented minorities in science, technology, engineering, and mathematics (STEM). 
I cannot know for certain that a research project will have that result, but I can 
know some aspects of a research plan that are likely to do so. If my research plan, 
for instance, includes some form of collaboration with colleagues from an HBCU, 
then it is more likely to expand the participation of underrepresented minorities 
than if it does not include that collaboration. Of course the collaboration is not a 
failsafe guarantee what the broader impact will be, but it increases the likelihood — 
just as certain elements of the scientific research plan will increase the likelihood 
of certain experimental outcomes. 

Q2. Having sat on review panels, please tell me how heavily the broader impacts of 
a proposal are weighed when rating the proposals you were considering. Did the eval- 
uation of broader impacts have any effect on the evaluation of the intellectual merit 
of the proposals? 

A2. Thank you. Chairman Brooks, for that question. As Dr. Marrett noted, the 
NSF has many processes in place to guard against bias within review committees — 
to make sure, in other words, that as far as is possible, the process identifies the 
projects with the most significant intellectual merit. Ideally, questions of broader 
impact are used only to decide afterwards between proposals of equal intellectual 
merit. Certainly at Indiana University, when we are allocating resources, we make 
every effort to do the same thing — to remove conflicts of interest, to focus on the 
intellectual merit of competing proposals, and to leave “broader impact” and stra- 
tegic considerations for later. 

Realistically, of course, it is impossible to guard against all variables that might 
prioritize some other aspect of a proposal over its intellectual merit. Indeed, one 
might worry that the expansion of broader impact goals as laid out in the America 
Competes Act increases that difficulty. But my experience is that more often than 
not, the intellectual merit of a proposal is the issue that is most determinative in 
funding recommendations, since consideration of broader impacts is generally a sec- 
ond-level concern. 
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